Memory management method and system for neural network model, and electronic device

Abstract

The present invention provides a memory management method and system for a neural network model, and an electronic device, and relates to the technical field of neural network learning. The memory management method includes: constructing a target directed graph of a neural network model, the target directed graph includes multiple operators and multiple directed edges; obtaining a continuous memory area from a target device; partitioning the continuous memory area to obtain multiple memory partitions, the multiple memory partitions include at least a first memory partition, a second memory partition, and a third memory partition; and pre-allocating memory for execution data of operators in the target directed graph based on the multiple memory partitions. The memory management method provided by the present invention can reduce the overhead of system functions that call device memory to allocate or release memory, and can improve the reasoning speed and performance of the neural network model.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network learning, and in particular to a memory management method and system for a neural network model, and an electronic device. Background Art

[0002] With the continuous development of science and technology, neural network learning has also made breakthrough progress in computer vision, image or speech recognition, natural language processing and other fields. In the application of neural network models, more and more neural network learning algorithms need to be deployed on resource-constrained embedded devices.

[0003] However, the inventors found that large models (neural network models with large-scale parameters and complex computing structures) will be affected by computing power and DDR (Double Data Rate) storage in resource-constrained embedded devices, which makes neural network models with more layers face some technical challenges in application in resource-constrained embedded devices.

[0004] For example, a neural network model with more layers requires a lot of computing resources, resulting in slower inference speed; and a deep neural network model requires more memory to store parameters and intermediate calculation results. However, due to the limited memory of embedded devices, there is a problem that large models may not be accommodated. Summary of the invention

[0005] The present invention provides a memory management method and system for a neural network model, and an electronic device, which are used to solve the problem of limited memory of an embedded device during operator execution of a neural network model.

[0006] According to one aspect of the present invention, the present invention provides a memory management method for a neural network model, which memory management method includes: constructing a target directed graph of the neural network model, the target directed graph including multiple operators and multiple directed edges; obtaining a continuous memory area from a target device; performing partition management on the continuous memory area to obtain multiple memory partitions, the multiple memory partitions including at least a first memory partition, a second memory partition and a third memory partition; and pre-allocating memory for execution data of operators in the target directed graph based on the multiple memory partitions.

[0007] According to some embodiments of the present invention, the execution data of an operator includes parameter data, and pre-allocating memory for the execution data of an operator in a target directed graph based on multiple memory partitions includes: determining the size of all parameter data of the operator during the execution of the operator; determining the total required memory based on the size of all parameter data; and allocating a continuous memory area in the first memory partition accordingly based on the total required memory.

[0008] According to some embodiments of the present invention, the memory management method also includes: re-arranging the parameter data stored in the first memory partition to save relevant parameter information of the parameter data to the first memory partition; saving the memory offset value of the relevant parameter information in the first memory partition to an external storage area, so that the external storage area contains information related to the position of the parameter data in the first memory partition.

[0009] According to some embodiments of the present invention, the execution data of the operator includes input or output data, and pre-allocating memory for the execution data of the operator in the target directed graph based on multiple memory partitions includes: allocating a first memory with a first preset byte based on the input or output data; performing memory alignment on the first memory to obtain a second memory; determining whether there is a minimum memory that satisfies the second memory based on managing the free memory blocks of the third memory partition; in the case where there is a minimum memory that satisfies the second memory, determining the minimum memory in the free memory blocks; judging whether the difference between the minimum memory and the second memory is greater than a first preset threshold; in the case where the difference between the minimum memory and the second memory is greater than the first preset threshold, dividing the minimum memory to add the divided memory area to the free memory block; adding the memory area allocated to the minimum memory to the used memory block; in the case where the difference between the minimum memory and the second memory is not greater than the first preset threshold, directly adding the memory area allocated to the minimum memory to the used memory block.

[0010] According to some embodiments of the present invention, when there is no minimum memory that satisfies the second memory, the memory management method also includes: calling a preset interface to obtain expanded memory from the target device; inputting memory information of the expanded memory into a preset hash table; judging whether the difference between the expanded memory and the second memory is greater than a second preset threshold; when the difference between the expanded memory and the second memory is greater than the second preset threshold, dividing the expanded memory into the size of the second memory, and adding the divided memory area to the free memory block; adding the divided memory area of ​​the second memory to the used memory block; when the difference between the expanded memory and the second memory is not greater than the second preset threshold, directly adding the divided memory area of ​​the second memory to the used memory block.

[0011] According to some embodiments of the present invention, a memory management method includes: releasing the input data of an operator in a target directed graph based on multiple memory partitions, including: determining the reference count of the input data of the operator; decrementing the reference count unitarily after the operator completes execution; traversing the reference count of the operator, releasing the input data with a reference count of 0, and returning the released memory to a third memory partition.

[0012] According to some embodiments of the present invention, a memory management method includes: performing memory optimization on the output data of a first type of operator in a target directed graph based on multiple memory partitions, including: directly reusing the memory of the input data of the first type of operator for the output data of the first type of operator; wherein the first type of operator is an operator type whose output data directly comes from the input data.

[0013] According to some embodiments of the present invention, a memory management method includes: performing memory optimization on the output data of a second type operator in a target directed graph based on multiple memory partitions, including: allocating a total memory to the output data of the second type operator and the output data of all other operators; wherein the second type operator is an operator type whose input data comes from the output data of multiple operators.

[0014] According to another aspect of the present invention, the present invention provides a memory management system for a neural network model, including a graph construction module, a memory processing module and a memory management module. The graph construction module constructs a target directed graph of the neural network model, and the target directed graph includes multiple operators and multiple directed edges; the memory processing module obtains a continuous memory area from the target device, and partitions the continuous memory area to obtain multiple memory partitions, and the multiple memory partitions include at least a first memory partition, a second memory partition and a third memory partition; the memory management module pre-allocates memory for execution data of operators in the target directed graph based on the multiple memory partitions.

[0015] According to another aspect of the present invention, the present invention further provides an electronic device. The electronic device includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors can implement the memory management method described above.

[0016] According to another aspect of the present invention, the present invention further provides a non-volatile computer-readable storage medium having a computer program stored thereon, which can implement the memory management method described above when executed by a processor.

[0017] The present invention constructs a target directed graph of a neural network model, and partitions the memory pool of the target directed graph during the execution of an operator, pre-allocates corresponding memory areas according to the execution data of the operator, and can apply for and release memory from a specific memory area during the execution of the operator, thereby reducing the overhead of system functions that frequently call device memory for memory allocation / release, thereby improving the reasoning speed and performance of the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram showing an existing memory allocation;

[0020] Figure 2 A schematic diagram showing a memory management method according to an embodiment of the present invention;

[0021] Figure 3 A schematic diagram showing an operator node of an embodiment of the present invention;

[0022] Figure 4 A schematic diagram showing directed edges according to an embodiment of the present invention;

[0023] Figure 5 A schematic diagram showing a target directed graph according to an embodiment of the present invention;

[0024] Figure 6 A schematic diagram showing multiple memory partitions according to an embodiment of the present invention;

[0025] Figure 7 Another schematic diagram showing the memory management method according to an embodiment of the present invention;

[0026] Figure 8 Another schematic diagram showing multiple memory partitions according to an embodiment of the present invention;

[0027] Fig. 9 Another schematic diagram showing the memory management method according to an embodiment of the present invention;

[0028] Fig.10 Another schematic diagram showing the memory management method according to an embodiment of the present invention;

[0029] Fig.11 Another schematic diagram showing the memory management method according to an embodiment of the present invention;

[0030] Fig.12 Another schematic diagram showing the memory management method according to an embodiment of the present invention;

[0031] Fig.13 Another schematic diagram showing the memory management method according to an embodiment of the present invention;

[0032] Fig.14 Another schematic diagram showing the memory management method according to an embodiment of the present invention;

[0033] Fig.15A schematic diagram showing memory optimization of the first type of operator according to an embodiment of the present invention is shown;

[0034] Fig.16 A schematic diagram showing a Layer according to an embodiment of the present invention;

[0035] Fig.17 Another schematic diagram showing the memory management method according to an embodiment of the present invention;

[0036] Fig.18 A schematic diagram showing memory optimization of the second type of operator according to an embodiment of the present invention is shown;

[0037] Fig.19 A schematic diagram showing a memory management system according to an embodiment of the present invention.

[0038] Description of reference numerals:

[0039] Memory management system 1; graph construction module 10; memory processing module 20; memory management module 30. DETAILED DESCRIPTION

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0041] The described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of these specific details, or other modes, components, materials, devices, etc. may be adopted. In these cases, known structures, methods, devices, implementations, materials or operations will not be shown or described in detail.

[0042] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0043] The terms "first", "second" and the like in the specification, claims and the above drawings of the present invention are used to distinguish different objects rather than to describe a specific order.

[0044] The following is a clear and complete description of the technical solution of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0045] The full names of the English abbreviations involved in the present invention and their Chinese meanings are:

[0046] DDR: Double Data Rate, double data transmission rate;

[0047] Conv2d_8: convolutional layer;

[0048] conv2d_weight: weight parameter of the convolutional layer;

[0049] conv2d_bias: bias parameter of the convolutional layer;

[0050] BatchNorm: Batch normalization, referred to as BN;

[0051] BN_weight: BatchNorm weight parameter;

[0052] I2F: Int convert float, integer floating point conversion;

[0053] Static DDR memory: static DDR memory;

[0054] Dynamic DDR memory: Dynamic DDR memory;

[0055] ID: Identity document, identity identification;

[0056] Const operator: constant operator;

[0057] Reshape operator: deformation operator;

[0058] Conv2d operator: two-dimensional convolution operator;

[0059] POOL2D operator: two-dimensional pooling layer operator;

[0060] Relu operator: activation operator;

[0061] Concat operator: connection operator;

[0062] Best filt: best adaptive algorithm;

[0063] data_buffe_: data buffer.

[0064] Memory management is an important part of neural network model reasoning. When performing neural network model reasoning in resource-constrained embedded devices, it is necessary to reduce the memory occupied by computing resources as much as possible. At present, the memory occupied by neural network model reasoning generally includes the following three aspects:

[0065] 1. Weight parameters, such as parameters in neural network models;

[0066] 2. Operator input / output, such as the input and output of each layer of operators in the neural network model;

[0067] 3. Other aspects, such as temporary memory and some resident memory required during the reasoning process.

[0068] At present, the existing technology generally reduces the memory occupied by weight parameters through model compression (such as quantization and pruning operations, etc.); and the existing technology generally reduces the memory occupied by operator input / output through compression coding or memory reuse.

[0069] For example, Figure 1 A schematic diagram of an existing memory allocation is shown. Figure 1 As shown in the figure, first allocate DDR memory to the input data input0 of the Conv2d_8 (convolutional layer) operator, and write the target inference data into the DDR, then allocate the DDR memory of the input parameters conv2d_weight (weight parameter of the convolutional layer) and conv2d_bias (bias parameter of the convolutional layer) of the Conv2d_8 operator, read the parameter data into the corresponding DDR, and then allocate the DDR memory of output1 of the Conv2d_8 operator, and then execute the Conv2d_8 operator. After the execution is completed, write the result into the DDR memory of output1, and release the DDR memory of input0, conv2d_weight and conv2d_bias.

[0070] like Figure 1As shown in the figure, since the input of the BatchNorm (batch normalization, BN for short) operator reuses the DDR memory of the output output1 of the Conv2d_8 operator, repeated memory application and redundant memory copying can be avoided. The DDR memory of the input parameter BN_weight (BN weight parameter) of the BatchNorm operator is allocated, and the data is read into this DDR memory. Then the DDR memory of the output output3 of the BatchNorm operator is allocated, and then the BatchNorm operator is executed. After the execution is completed, the result is written to the DDR memory of output3, and the DDR memory of BN_weight is released.

[0071] The input of the operator I2F (Int convert float) reuses the DDR memory of the output output1 of the Conv2d_8 operator, allocates the DDR memory of the output output2 of the I2F operator, and then executes the I2F operator. After the execution is completed, the result is written to the DDR memory of output2.

[0072] The above reasoning process can obtain the reasoning results output2 and output3. However, the inventors found that in the above reasoning process, it is necessary to apply for and release DDR memory, and frequent application and release of DDR memory will reduce the performance of the neural network model. For embedded devices with strict performance requirements, this will reduce the performance of the device, and even cause large models to fail to run due to limited DDR memory.

[0073] According to one aspect of the present invention, the present invention provides a memory management method for a neural network model.

[0074] Figure 2 A schematic diagram of a memory management method according to an embodiment of the present invention is shown. Figure 2 As shown, the memory management method includes steps S100 to S400. Exemplarily, the memory management method may be executed by a memory management system having computing capabilities.

[0075] According to an example embodiment, in step S100, the memory management system constructs a target directed graph of the neural network model, and the target directed graph includes multiple operators and multiple directed edges.

[0076] A directed graph refers to a graph structure in which the connections between operators (or operator nodes) in a neural network model are directional. A directed graph consists of a set of vertices (operator nodes) and a set of directed edges. Vertices are used to represent a type of operator nodes that need to be executed during the reasoning process of the neural network model (which can be represented by the OpLayer class); directed edges are directed line segments connecting two operators, which are used to represent the input relationship or output relationship between the two operators (generally represented by the direction of the arrow, which can be represented by the Layer class).

[0077] The memory management system responds to the user's construction instructions, creates multiple different operator nodes (OpLayer), and then builds the target directed graph (ie, DAGraph) by connecting the directed edges between the operator nodes.

[0078] Figure 3 A schematic diagram showing an operator node of an embodiment of the present invention; Figure 4 A schematic diagram showing directed edges according to an embodiment of the present invention; Figure 5 A schematic diagram showing a target directed graph according to an embodiment of the present invention.

[0079] like Figure 3 As shown in the figure, OpLayer is described as a type of operator node, such as Const operator (constant operator), Reshape operator (deformation operator), Conv2d operator (two-dimensional convolution operator) and Relu operator (activation operator). A few types of operators have no input data, but all operators have output data.

[0080] like Figure 4 As shown, Layer describes the structure of the input data and output data of the operator, including the size information of the data, the storage address information, the dimension information of the data, the type information of the data, and the input or output data of the operator.

[0081] like Figure 5 As shown, the final DAGraph data structure is constructed by OpLayer and Layer. Each operator has a corresponding ID, through which the corresponding operator object can be found. Each operator object has an input name or an output name, and the input object or output object of the operator can be found through the input name or output name.

[0082] As an embodiment of the present invention, DAGraph may include three operator nodes, such as Conv2d_8 operator, BatchNorm operator and I2F operator.

[0083] The Conv2d_8 operator includes two types of input data, input data input0 and two additional input parameters conv2d_weight and conv2d_bias. The output of the Conv2d_8 operator is output1.

[0084] The input data of the BatchNorm operator reuses the output1 of the Conv2d_8 operator and an additional input parameter BN_weight. The output of the BatchNorm operator is output3.

[0085] The input data of the I2F operator is output1 of the reused Conv2d_8 operator, and the output of the I2F operator is output2.

[0086] In step S200, the memory management system obtains a continuous memory area from the target device.

[0087] In step S300, the memory management system performs partition management on the continuous memory area to obtain a plurality of memory partitions, which at least include a first memory partition, a second memory partition and a third memory partition.

[0088] For example, the target device may be an embedded device, and the continuous memory area may be a DDR memory. The memory management system partitions the continuous DDR memory and divides it into three partitions, such as a first memory partition, a second memory partition, and a third memory partition.

[0089] In step S400, the memory management system pre-allocates memory for operator execution data in the target directed graph based on multiple memory partitions.

[0090] Figure 6 A schematic diagram showing multiple memory partitions according to an embodiment of the present invention.

[0091] According to an example embodiment, the operator execution data includes parameter data, temporarily stored data, and input or output data, among others.

[0092] According to an example embodiment, Figure 6 As shown, the first memory partition (which can be Static DDR memory) can be used to store parameter data such as weights and biases related to the operator; the second memory partition (which can be Others) can be used to store temporary storage data during the execution of the operator; the third memory partition (which can be DynamicDDR memory) can be used to store the input or output data of the operator.

[0093] Through the above embodiments, the present invention constructs a target directed graph of a neural network model, and partitions the memory pool of the target directed graph during the execution of the operator, pre-allocates the corresponding memory area according to the execution data of the operator, and can apply for and release memory from a specific memory area during the execution of the operator, thereby reducing the overhead of system functions that frequently call device memory for memory allocation / release, thereby improving the reasoning speed and performance of the neural network model.

[0094] Figure 7 Another schematic diagram showing the memory management method according to an embodiment of the present invention; Figure 8 Another schematic diagram showing multiple memory partitions according to an embodiment of the present invention; Fig. 9 Another schematic diagram showing the memory management method according to an embodiment of the present invention.

[0095] Alternatively, if Figure 7 As shown, in step S400, the execution data of the operator includes parameter data, and the memory management system pre-allocates memory for the execution data of the operator in the target directed graph based on multiple memory partitions, including steps S411-S413.

[0096] In step S411, the memory management system determines the size of all parameter data of the operator during the execution of the operator.

[0097] In step S412, the memory management system determines the total required memory according to the size of all parameter data.

[0098] In step S413, the memory management system allocates a corresponding continuous memory area in the first memory partition based on the total required memory.

[0099] For example, during the execution of the Conv2d_8 operator, the weight and bias parameter data of conv2d_weight and conv2d_bias are required. By counting the data size of all input parameters (such as conv2d_weight and conv2d_bias), the total memory required for storing the weight and bias parameter data of all operators can be obtained. Figure 8 As shown, the memory management system allocates a continuous DDR memory in the Static DDR memory partition to statically store parameter data such as weights and biases of the quantized and pruned operators.

[0100] Through the above embodiments, the present invention can perform partition memory management on the execution data during the execution of the operator by pre-allocating the parameter data of the operator to the corresponding static DDR storage.

[0101] Alternatively, if Fig. 9As shown, the memory management method provided by the present invention also includes steps S500-S600.

[0102] In step S500, the memory management system re-arranges the parameter data stored in the first memory partition to save the relevant parameter information of the parameter data to the first memory partition.

[0103] In step S600, the memory management system saves the memory offset value of the relevant parameter information in the first memory partition to the external storage area, so that the external storage area contains the information related to the position of the parameter data in the first memory partition.

[0104] According to the example embodiment, the memory management system reorders the parameter data stored in the static DDR storage, that is, according to the execution order of the operators in the target directed graph, the relevant parameter information (such as position-related information) of the operators with input data is sequentially saved to the static DDR storage. Then, the memory offset value of each parameter data in the static DDR storage is recorded and saved to the attr parameter of the operator, such as Figure 8 As shown, the attr parameter is saved in the OS memory area (operating system memory).

[0105] Exemplarily, the attr parameter is used to characterize the parameter information of the operator, such as the memory offset value, parameter size, and other information stored in the first memory partition of the parameter data.

[0106] For example, Figure 8 As shown, the input parameters conv2d_weight and conv2d_bias of the Conv2d_8 operator are stored in the static DDR storage, and the parameters describing other information of the Conv2d_8 operator are saved in the attr parameter, which contains information related to the location of conv2d_weight and conv2d_bias in the static DDR storage.

[0107] Through the above embodiments, the present invention is configured so that during the execution of the Conv2d_8 operator, the corresponding weights or bias parameters can be obtained through the position-related information without allocating DDR memory for weights and bias parameters, thereby reducing the time spent on reading data into the DDR memory, as well as reducing memory fragmentation and memory release time.

[0108] Fig.10 Another schematic diagram showing the memory management method according to an embodiment of the present invention.

[0109] Alternatively, if Fig.10As shown, in step S400, the execution data of the operator includes input or output data, and the memory management system pre-allocates memory for the execution data of the operator in the target directed graph based on multiple memory partitions, including steps S421-S423.

[0110] In step S421, the memory management system allocates a first memory having a first preset byte based on input or output data.

[0111] In step S422, the memory management system performs memory alignment on the first memory to obtain a second memory.

[0112] In step S423, the memory management system determines whether there is a minimum memory that satisfies the second memory based on managing the free memory blocks of the third memory partition.

[0113] For example, the memory management system allocates an m based on the input or output data of an operator. 0 Bytes of memory first, followed by Byte aligned memory, get the second memory m 1 The memory management system uses a preset algorithm (such as Best filt algorithm, best adaptive algorithm) in a free memory block (such as free_block hash table) to find out whether there is a minimum memory that satisfies the second memory.

[0114] As an example, assuming that the second memory is 400M, if there are memory blocks of 500M, 600M and 700M respectively, the smallest memory block (i.e. 500M) is determined as the minimum memory that satisfies the second memory. This setting can make it possible to find the minimum memory area that meets the requirements as much as possible, thereby maximizing memory utilization.

[0115] like Fig.10 As shown, if there is a minimum memory block that satisfies the second memory, proceed to step S424; otherwise, proceed to step S425.

[0116] Fig.11 Another schematic diagram showing the memory management method according to an embodiment of the present invention is shown as follows: Fig.11 As shown, step S424 also includes steps S4241-S4244.

[0117] In step S4241, the memory management system determines the minimum memory in the free memory block.

[0118] In step S4242, the memory management system determines whether the difference between the minimum memory and the second memory is greater than a first preset threshold.

[0119] In step S4243, when the difference between the minimum memory and the second memory is greater than a first preset threshold, the memory management system divides the minimum memory and merges the divided memory areas into the free memory blocks.

[0120] In step S4244, the memory management system adds the memory area allocated to the minimum memory to the used memory block.

[0121] In step S4242, when the memory management system determines that the difference between the minimum memory and the second memory is not greater than the first preset threshold, it directly enters step S4244.

[0122] For example, the minimum memory is b 0 , the memory management system determines b 0 With m 1 Is the difference greater than the first preset threshold value T 1 , in b 0 -m 1 >T 1 In the case of 0 The memory area is split and added to the free_block hash table. The allocated memory block is then added to the used memory block (such as the used_block hash table).

[0123] The memory management system is in b 0 -m 1 ≤T 1 In the case of , the allocated memory block is directly added to the used memory block (such as the used_block hash table).

[0124] Through the above embodiments, the present invention can quickly search for the minimum memory that meets the requirements in the free memory blocks in the dynamic DDR storage through memory block alignment, the use of a hash table based on the Best Filt algorithm and a preset memory allocation scheme, thereby speeding up the allocation and release of memory and reducing the occurrence of memory fragmentation.

[0125] Fig.12 Another schematic diagram showing the memory management method according to an embodiment of the present invention is shown as follows: Fig.12 As shown, step S425 also includes steps S4241-S4245.

[0126] In step S4251, the memory management system calls a preset interface to obtain the expanded memory from the target device.

[0127] In step S4252, the memory management system inputs the memory of the expanded memory into a preset hash table.

[0128] For example, the memory management system calls the malloc interface (dynamic memory allocation interface) of the embedded device to obtain the expanded memory (such as DDR_1) from the DDR memory. Then, the memory information of DDR_1 is input into a preset hash table (such as the ddr_total_mem hash table, which is used to record the allocated memory information).

[0129] In step S4253, the memory management system determines whether the difference between the expanded memory and the second memory is greater than a second preset threshold.

[0130] In step S4254, when the difference between the expanded memory and the second memory is greater than a second preset threshold, the memory management system divides the expanded memory into the second memory size and adds the divided memory area to the free memory block.

[0131] In step S4255, the memory management system adds the divided memory area of ​​the second memory to the used memory block.

[0132] In step S4253, when the memory management system determines that the difference between the expanded memory and the second memory is not greater than the second preset threshold, the process directly proceeds to step S4255.

[0133] For example, the memory management system determines whether DDR_1 and m 1 Is the difference greater than the second preset threshold value T 2 , in DDR_1-m 1 >T 2 In the case of, after the memory management system divides the expanded memory into the second memory size, the divided memory area is added to the free_block, and then the divided memory area of ​​the second memory is added to the used memory block (used_block hash table).

[0134] Memory management system in DDR_1-m 1 ≤T 2 In this case, the memory area of ​​the split second memory is directly added to the used memory block (used_block hash table).

[0135] Through the above embodiments, when the present invention fails to find the minimum memory block that meets the requirements in the free memory blocks in the dynamic DDR storage, the memory expansion mechanism is triggered to perform automatic expansion (e.g., 100M can be automatically expanded each time) until the memory allocation requirements are met.

[0136] Fig.13 Another schematic diagram of the memory management method of the embodiment of the present invention is shown. The memory management method provided by the present invention also includes step S700, such as Fig.13 As shown, step S700 may also include steps S710-S730.

[0137] In step S710 , the memory management system determines the reference count of the input data of the operator.

[0138] In step S720, after executing the operator, the memory management system decrements the reference count in units.

[0139] In step S730, the memory management system traverses the reference counts of the operators, releases the input data with a reference count of 0, and returns the released memory to the third memory partition.

[0140] For example, the memory management system first counts the reference counts of the input data of all operators, and executes the operators in sequence according to the operator execution order of the target directed graph.

[0141] During the execution of the operator, the DDR memory of each input data and each output data of the operator is allocated. If the name of the input data of the operator is the same as the name of the output data of the operator in the previous layer, the DDR memory of the output data of the previous layer is directly reused. If the algorithm also requires parameters such as weights and biases, the memory content of each parameter is obtained in the static DDR storage according to the attr attribute value of the operator, and then the operator is executed and the corresponding results are written to the DDR memory of the output data.

[0142] After the operator is executed, the reference count of all input data of the operator is -1. When the reference count of all input data of the operator is 0, the DDR memory of the operator is released and the memory is returned to the dynamic DDR storage, completing the memory release of the operator.

[0143] Fig.14 Another schematic diagram showing the memory management method according to an embodiment of the present invention. Fig.15 A schematic diagram showing memory optimization of the first type of operator according to an embodiment of the present invention is shown; Fig.16 A schematic diagram of Layer according to an embodiment of the present invention is shown.

[0144] Optionally, the memory management method provided by the present invention further includes step S800.

[0145] In step S800, the memory management system performs memory optimization on the output data of the first type of operators in the target directed graph based on multiple memory partitions, including: the memory management system directly reuses the memory of the input data of the first type of operators for the output data of the first type of operators.

[0146] According to an example embodiment, the first type of operator is an operator type where output data comes directly from input data.

[0147] Fig.15Figure (1) shows the Reshape operator before memory optimization. Fig.15 Figure (2) shows the Reshape operator after memory optimization.

[0148] For example, as shown in Figure (1), the Reshape operator has the characteristic of only changing the dimension but not the data, and its output data comes directly from the input data. As shown in Figure (2), by reusing the DDR memory of the input data with the output data of the Reshape operator, the allocation of DDR memory and the time consumption of copying can be reduced, thereby reducing the overall time consumption of neural network model inference.

[0149] Alternatively, if Fig.16 As shown, the memory management system can add data_buffer_ (data buffer) in the input data layer to manage the actual data. Fig.16 As shown, when the Reshape operator is executed, after the reused input data is output, the reference count of data_buffe_ in the input data layer is +1. After one operator is executed, the reference count of the input data layer is -1. When the reference count of the input data layer is 0, if the memory management system determines that data_buffe_ is not reused by other layers, the DDR memory of the input data layer is released.

[0150] Alternatively, if Fig.17 As shown, the memory management method provided by the present invention also includes step S900.

[0151] In step S900, the memory management system performs memory optimization on the output data of the second type operator in the target directed graph based on multiple memory partitions, including: the memory management system allocates a total memory to the output data of the second type operator and the output data of all other operators.

[0152] According to an example embodiment, the second type operator is an operator type whose input data comes from output data of a plurality of operators.

[0153] Fig.18 Figure (3) shows the Concat operator before memory optimization. Fig.18 Figure (4) shows the Concat operator after memory optimization.

[0154] For example, as shown in Figure (3), the Concat operator has the characteristic of concatenating multiple input data into a new data. The memory management system extracts the execution order of the Concat operator, allocates the total DDR memory of the output data of the Concat operator, and continues to execute the operator that needs to be concatenated.

[0155] The Concat operator concatenates multiple input data (i.e., the output data output1 and output2 of the POOL2D operator and the RELU operator), and saves the concatenated data to the output data output3 of the Concat operator. As shown in Figure (4), the memory management system pre-allocates a DDR memory to the output data output1 of the Concat operator, and also sets the output data of the POOL2D operator and the RELU operator to the DDR memory.

[0156] Through the above embodiments, the present invention is configured in this way, which can reduce the allocation and release of DDR memory of POOL2D operator and RELU operator, reduce the time consumption of at least two data copies, and thus improve the reasoning speed of the neural network model.

[0157] According to another aspect of the present invention, the present invention provides a memory management system for a neural network model. Fig.19 As shown, the memory management system 1 includes a graph construction module 10 , a memory processing module 20 and a memory management module 30 .

[0158] According to an example embodiment, the graph construction module 10 constructs a target directed graph of a neural network model, the target directed graph including a plurality of operators and a plurality of directed edges.

[0159] A directed graph refers to a graph structure in which the connections between operators (or operator nodes) in a neural network model are directional. A directed graph consists of a set of vertices (operator nodes) and a set of directed edges. Vertices are used to represent a type of operator nodes that need to be executed during the reasoning process of the neural network model (which can be represented by the OpLayer class); directed edges are directed line segments connecting two operators, which are used to represent the input relationship or output relationship between the two operators (generally represented by the direction of the arrow, which can be represented by the Layer class).

[0160] The graph construction module 10 responds to the user's construction instructions, creates multiple different operator nodes (OpLayer), and then constructs the target directed graph (ie, DAGraph) by connecting the directed edges between the operator nodes.

[0161] OpLayer is described as a type of operator node, such as const operator (constant operator), reshape operator (deformation operator), conv2d operator (two-dimensional convolution operator) and relu operator (activation operator). A few types of operators have no input data, but all operators have output data.

[0162] Layer describes the structure of the input data and output data of the operator, including data size information, storage address information, data dimension information, data type information, and operator input or output data.

[0163] The final DAGraph data structure is constructed by OpLayer and Layer. Each operator has a corresponding ID, through which the corresponding operator object can be found. Each operator object has an input name or output name, and the input object or output object of the operator can be found through the input name or output name.

[0164] As an embodiment of the present invention, DAGraph may include three operator nodes, such as Conv2d_8 operator, BatchNorm operator and I2F operator.

[0165] The Conv2d_8 operator includes two types of input data, input data input0 and two additional input parameters conv2d_weight and conv2d_bias. The output of the Conv2d_8 operator is output1.

[0166] The input data of the BatchNorm operator reuses the output1 of the Conv2d_8 operator and an additional input parameter BN_weight. The output of the BatchNorm operator is output3.

[0167] The input data of the I2F operator is output1 of the reused Conv2d_8 operator, and the output of the I2F operator is output2.

[0168] According to an example embodiment, the memory processing module 20 obtains a continuous memory area from the target device.

[0169] The memory processing module 20 performs partition management on the continuous memory area to obtain a plurality of memory partitions, which at least include a first memory partition, a second memory partition and a third memory partition.

[0170] For example, the target device may be an embedded device, and the continuous memory area may be a DDR memory. The memory processing module 20 performs partition management on the continuous DDR memory and divides it into three partitions, such as a first memory partition, a second memory partition, and a third memory partition.

[0171] According to an example embodiment, the memory management module 30 performs memory pre-allocation for operator execution data in the target directed graph based on a plurality of memory partitions.

[0172] According to an example embodiment, the operator execution data includes parameter data, temporarily stored data, and input or output data, among others.

[0173] The first memory partition (which can be Static DDR memory) can be used to store parameter data such as weights and biases related to the operator; the second memory partition (which can be Others) can be used to store temporary storage data during the execution of the operator; the third memory partition (which can be Dynamic DDR memory) can be used to store the input or output data of the operator.

[0174] Through the above embodiments, the present invention constructs a target directed graph of a neural network model, and partitions the memory pool of the target directed graph during the execution of the operator, pre-allocates the corresponding memory area according to the execution data of the operator, and can apply for and release memory from a specific memory area during the execution of the operator, thereby reducing the overhead of system functions that frequently call device memory for memory allocation / release, thereby improving the reasoning speed and performance of the neural network model.

[0175] According to an example embodiment, the memory management module 30 determines the size of all parameter data of an operator during the execution of the operator.

[0176] The memory management module 30 determines the total required memory according to the size of all parameter data.

[0177] The memory management module 30 allocates a corresponding continuous memory area in the first memory partition based on the total required memory.

[0178] For example, during the execution of the operator Conv2d_8, the weight and bias parameter data of conv2d_weight and conv2d_bias are required. By counting the data size of all input parameters (such as conv2d_weight and conv2d_bias), the total memory required for storing the weight and bias parameter data of all operators can be obtained. The memory management module 30 allocates a continuous DDR memory in the Static DDR memory partition to statically store the weight and bias parameter data of the quantized and pruned operators.

[0179] Through the above embodiments, the present invention can perform partition memory management on the execution data during the execution of the operator by pre-allocating the parameter data of the operator to the corresponding static DDR storage.

[0180] According to an example embodiment, the memory management module 30 re-arranges the parameter data stored in the first memory partition to save the relevant parameter information of the parameter data to the first memory partition.

[0181] The memory management module 30 saves the memory offset value of the relevant parameter information in the first memory partition to the external storage area, so that the external storage area contains the information related to the position of the parameter data in the first memory partition.

[0182] According to the exemplary embodiment, the memory management module 30 reorders the parameter data stored in the static DDR storage, that is, according to the execution order of the operators in the target directed graph, the relevant parameter information (such as position-related information) of the operators with input data is sequentially saved in the static DDR storage. Then, the memory offset value of each parameter data in the static DDR storage is recorded and saved in the attr parameter of the operator, and the attr parameter is saved in the OS memory area (operating system memory).

[0183] Exemplarily, the attr parameter is used to characterize the parameter information of the operator, such as the memory offset value, parameter size, and other information stored in the first memory partition of the parameter data.

[0184] Exemplarily, the input parameters conv2d_weight and conv2d_bias of the Conv2d_8 operator are stored in the static DDR storage, and the parameters describing other information of the Conv2d_8 operator are saved in the attr parameter, which includes information related to the position of conv2d_weight and conv2d_bias in the static DDR storage.

[0185] Through the above embodiments, the present invention is configured so that during the execution of the Conv2d_8 operator, the corresponding weights or bias parameters can be obtained through the position-related information without allocating DDR memory for weights and bias parameters, thereby reducing the time spent on reading data into the DDR memory, as well as reducing memory fragmentation and memory release time.

[0186] Optionally, the memory management module 30 allocates a first memory having a first preset byte based on input or output data.

[0187] The memory management module 30 performs memory alignment on the first memory to obtain the second memory.

[0188] The memory management module 30 determines whether there is a minimum memory that satisfies the second memory based on managing the free memory blocks of the third memory partition.

[0189] For example, the memory management module 30 allocates an m based on the input or output data of the operator. 0 Bytes of memory first, followed by Byte aligned memory, get the second memory m 1The memory management module 30 uses a preset algorithm (such as Best filt algorithm, best adaptive algorithm) in a free memory block (such as free_block hash table) to find whether there is a minimum memory that satisfies the second memory.

[0190] As an example, assuming that the second memory is 400M, if there are memory blocks of 500M, 600M and 700M respectively, the smallest memory block (i.e. 500M) is determined as the minimum memory that satisfies the second memory. This setting can make it possible to find the minimum memory area that meets the requirements as much as possible, thereby maximizing memory utilization.

[0191] Optionally, the memory management module 30 determines the minimum memory in the free memory blocks.

[0192] The memory management module 30 determines whether the difference between the minimum memory and the second memory is greater than a first preset threshold.

[0193] When the difference between the minimum memory and the second memory is greater than a first preset threshold, the memory management module 30 divides the minimum memory and merges the divided memory areas into the free memory blocks.

[0194] The memory management module 30 adds the memory area allocated to the minimum memory to the used memory block.

[0195] For example, the minimum memory is b 0 , the memory management module 30 determines b 0 With m 1 Is the difference greater than the first preset threshold value T 1 , in b 0 -m 1 >T 1 In the case of 0 The memory area is split and added to the free_block hash table. The allocated memory block is then added to the used memory block (such as the used_block hash table).

[0196] Memory management module 30 in b 0 -m 1 ≤T 1 In the case of , the allocated memory block is directly added to the used memory block (such as the used_block hash table).

[0197] Through the above embodiments, the present invention can quickly search for the minimum memory that meets the requirements in the free memory blocks in the dynamic DDR storage through memory block alignment, the use of a hash table based on the Best Filt algorithm and a preset memory allocation scheme, thereby speeding up the allocation and release of memory and reducing the occurrence of memory fragmentation.

[0198] Optionally, the memory management module 30 calls a preset interface to obtain the expanded memory from the target device.

[0199] The memory management module 30 inputs the memory of the expanded memory into a preset hash table.

[0200] For example, the memory management module 30 calls the malloc interface (dynamic memory allocation interface) of the embedded device to obtain the expanded memory (such as DDR_1) from the DDR memory, and then inputs the memory information of DDR_1 into a preset hash table (such as the ddr_total_mem hash table, which is used to record the allocated memory information).

[0201] The memory management module 30 determines whether the difference between the expanded memory and the second memory is greater than a second preset threshold.

[0202] When the difference between the expanded memory and the second memory is greater than a second preset threshold, the memory management module 30 divides the expanded memory into the second memory size and adds the divided memory area to the free memory block.

[0203] The memory management module 30 adds the divided memory area of ​​the second memory to the used memory block.

[0204] For example, the memory management module 30 determines whether DDR_1 and m 1 Is the difference greater than the second preset threshold value T 2 , in DDR_1-m 1 >T 2 In this case, after the memory management module 30 divides the expanded memory into the second memory size, the divided memory area is added to the free_block, and then the divided memory area of ​​the second memory is added to the used memory block (used_block hash table).

[0205] Memory management module 30 in DDR_1-m 1 ≤T 2 In this case, the memory area of ​​the split second memory is directly added to the used memory block (used_block hash table).

[0206] Through the above embodiments, when the present invention fails to find the minimum memory block that meets the requirements in the free memory blocks in the dynamic DDR storage, the memory expansion mechanism is triggered to perform automatic expansion (e.g., 100M can be automatically expanded each time) until the memory allocation requirements are met.

[0207] Optionally, the memory management module 30 determines a reference count of input data of an operator.

[0208] After executing the operator, the memory management module 30 decrements the reference count in units.

[0209] The memory management module 30 traverses the reference counts of the operators, releases the input data with a reference count of 0, and returns the released memory to the third memory partition.

[0210] For example, the memory management module 30 first counts the reference counts of input data of all operators, and executes the operators in sequence according to the operator execution order of the target directed graph.

[0211] During the execution of the operator, the DDR memory of each input data and each output data of the operator is allocated. If the name of the input data of the operator is the same as the name of the output data of the operator in the previous layer, the DDR memory of the output data of the previous layer is directly reused. If the algorithm also requires parameters such as weights and biases, the memory content of each parameter is obtained in the static DDR storage according to the attr attribute value of the operator, and then the operator is executed and the corresponding results are written to the DDR memory of the output data.

[0212] After the operator is executed, the reference count of all input data of the operator is -1. When the reference count of all input data of the operator is 0, the DDR memory of the operator is released and the memory is returned to the dynamic DDR storage, completing the memory release of the operator.

[0213] Optionally, the memory management module 30 performs memory optimization on the output data of the first type of operators in the target directed graph based on multiple memory partitions, including: the memory management module 30 directly reuses the memory of the input data of the first type of operators for the output data of the first type of operators.

[0214] According to an example embodiment, the first type of operator is an operator type where output data comes directly from input data.

[0215] For example, taking the Reshape operator as an example, the Reshape operator has the characteristic of only changing the dimension but not the data, and its output data comes directly from the input data. By reusing the DDR memory of the input data for the output data of the Reshape operator, the allocation of DDR memory and the time consumption of one copy can be reduced, thereby reducing the overall time consumption of neural network model inference.

[0216] Optionally, the memory management module 30 can add a data buffer in the input data layer to manage the actual data. Fig.16As shown, when the Reshape operator is executed, after the reused input data is output, the reference count of data_buffe_ in the input data layer is +1. After one operator is executed, the reference count of the input data layer is -1. When the reference count of the input data layer is 0, if the memory management module 30 determines that data_buffe_ is not reused by other layers, the DDR memory of the input data layer is released.

[0217] Optionally, the memory management module 30 performs memory optimization on the output data of the second type of operators in the target directed graph based on multiple memory partitions, including: the memory management module 30 allocates a total memory to the output data of the second type of operators and the output data of all other operators.

[0218] According to an example embodiment, the second type operator is an operator type whose input data comes from output data of a plurality of operators.

[0219] For example, taking the Concat operator as an example, the Concat operator has the characteristic of concatenating multiple input data into a new data. The memory management module 30 extracts the execution order of the Concat operator, allocates the total DDR memory of the output data of the Concat operator, and continues to execute the operator that needs to be concatenated.

[0220] The Concat operator concatenates multiple input data (i.e., output data output1 and output2 of the POOL2D operator and the RELU operator), and saves the concatenated data into output data output3 of the Concat operator. The memory management module 30 pre-allocates a DDR memory for the output data output1 of the Concat operator, and also sets the output data of the POOL2D operator and the RELU operator to the DDR memory.

[0221] Through the above embodiments, the present invention is configured in this way, which can reduce the allocation and release of DDR memory of POOL2D operator and RELU operator, reduce the time consumption of at least two data copies, and thus improve the reasoning speed of the neural network model.

[0222] According to another aspect of the present invention, an electronic device is provided, which includes: one or more processors; and a storage device for storing one or more programs, so that when the one or more programs are executed by the one or more processors, the one or more processors can implement the memory management method described above.

[0223] According to another aspect of the present invention, a non-volatile computer-readable storage medium is provided, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the memory management method described above can be implemented.

[0224] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions of the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A memory management method for a neural network model, characterized in that: include: Constructing a target directed graph of a neural network model, wherein the target directed graph includes a plurality of operators and a plurality of directed edges; Get a continuous memory area from the target device; Performing partition management on the continuous memory area to obtain a plurality of memory partitions, wherein the plurality of memory partitions include at least a first memory partition, a second memory partition, and a third memory partition; Pre-allocating memory for execution data of operators in the target directed graph based on the multiple memory partitions, where the execution data of the operators include input or output data, and pre-allocating memory for execution data of operators in the target directed graph based on the multiple memory partitions includes: Allocating a first memory having a first preset byte based on the input or output data; Performing memory alignment on the first memory to obtain a second memory; Determine whether there is a minimum memory that satisfies the second memory based on managing the free memory blocks of the third memory partition; In the case where there is a minimum memory that satisfies the second memory, determining the minimum memory in the free memory block; Determine whether the difference between the minimum memory and the second memory is greater than a first preset threshold; When the difference between the minimum memory and the second memory is greater than a first preset threshold, the minimum memory is divided to add the divided memory area to the free memory block; Adding the memory area allocated to the minimum memory to the used memory block; When the difference between the minimum memory and the second memory is not greater than a first preset threshold, the memory area allocated to the minimum memory is directly added to the used memory block.

2. The memory management method according to claim 1, characterized in that: The execution data of the operator includes parameter data, and the pre-allocating memory for the execution data of the operator in the target directed graph based on the multiple memory partitions includes: Determining the size of all parameter data of the operator during the execution of the operator; Determine the total required memory according to the size of all parameter data; Based on the total required memory, a continuous memory area is correspondingly allocated in the first memory partition.

3. The memory management method according to claim 2, characterized in that: The memory management method further includes: Rearranging the parameter data stored in the first memory partition to save relevant parameter information of the parameter data to the first memory partition; The memory offset value of the relevant parameter information in the first memory partition is saved in an external storage area, so that the external storage area contains information related to the position of the parameter data in the first memory partition.

4. The memory management method according to claim 1, characterized in that: In the case that there is no minimum memory satisfying the second memory, the memory management method further includes: Calling a preset interface to obtain the expanded memory from the target device; Inputting the memory information of the expanded memory into a preset hash table; Determine whether the difference between the expanded memory and the second memory is greater than a second preset threshold; When the difference between the expanded memory and the second memory is greater than a second preset threshold, the expanded memory is divided into the second memory size, and the divided memory area is added to the free memory block; adding the divided memory area of ​​the second memory to the used memory block; When the difference between the expanded memory and the second memory is not greater than the second preset threshold, the divided memory area of ​​the second memory is directly added to the used memory block.

5. The memory management method according to claim 1, characterized in that: The memory management method comprises: The memory is released for input data of operators in the target directed graph based on the multiple memory partitions, including: Determining a reference count of input data of the operator; After the operator is executed, the reference count is unit-decremented; The reference counts of the operators are traversed, the input data with a reference count of 0 is released from memory, and the released memory is returned to the third memory partition.

6. The memory management method according to claim 1, characterized in that: The memory management method comprises: Optimizing the memory of output data of the first type of operator in the target directed graph based on the multiple memory partitions includes: The output data of the first type of operator is directly reused in the memory of the input data of the first type of operator; The first type of operator is an operator type whose output data comes directly from input data.

7. The memory management method according to claim 1, characterized in that: The memory management method comprises: Optimizing the memory of the output data of the second type operator in the target directed graph based on the multiple memory partitions includes: Allocating the output data of the second type operator and the output data of all other operators to a total memory; The second type of operator is an operator type whose input data comes from the output data of multiple operators.

8. A memory management system for a neural network model, characterized in that: The memory management system is used to execute the memory management method according to any one of claims 1 to 7, and the memory management system includes: A graph construction module, constructing a target directed graph of a neural network model, wherein the target directed graph includes a plurality of operators and a plurality of directed edges; A memory processing module, which obtains a continuous memory area from a target device, and performs partition management on the continuous memory area to obtain a plurality of memory partitions, wherein the plurality of memory partitions include at least a first memory partition, a second memory partition, and a third memory partition; A memory management module performs memory pre-allocation for execution data of operators in the target directed graph based on the multiple memory partitions.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the memory management method as described in any one of claims 1-7.

Images