Method and computing device for optimizing the inference performance of neural network models

Abstract

The present invention provides a method and computing device for optimizing the reasoning performance of a neural network model, the method comprising: pre-running a directed graph structure of a neural network model; obtaining input data; and sequentially scheduling operators from the operator execution queue to perform reasoning calculations on the input data. The pre-running process comprises: sequentially obtaining each operator of each layer from the directed graph structure; determining the type of the operator, the type of the operator comprising: a constant operator, a deformation operator, a hardware operator, and a custom operator other than the above three operators; classifying the operator according to the type of the operator; and adding the processed operator to the operator execution queue according to the result of the classification process. The technical solution according to the present invention can optimize the loading process of the neural network model, shorten the model reasoning time, and reduce the model power consumption.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and computing device for optimizing the reasoning performance of a neural network model. Background Art

[0002] With the rapid development of deep learning, the application of neural networks in industry has made significant progress, such as analyzing historical data of equipment, predicting failures and maintenance needs, and automatically detecting production defects through image recognition technology. On restricted embedded products, the use of neural network models with many layers often faces many problems and challenges during actual loading and operation. Since deep neural networks require a lot of computing resources, the more network layers there are, the longer the model inference takes. Not only that, in general, the more complex the neural network model is, the more power it consumes during inference.

[0003] Figure 1 A schematic diagram of the traditional neural network model reasoning execution process is shown.

[0004] See also Figure 1 When performing artificial intelligence reasoning, traditional neural network models first start the runtime engine, load the model file from the model file, parse the execution graph structure and related parameter values, and then perform reasoning to obtain the final result. In traditional neural network model deployment solutions, large neural network models generally have loading problems and often fail to load. Since the entire model file must be loaded at one time, if the model file is large, there is often insufficient memory due to memory limitations of the operating system (OS) on limited embedded products, causing problems with model loading. In addition, deep large neural network models often take a long time to reason, resulting in high power consumption.

[0005] To this end, a technical solution is needed that can optimize the loading process of the neural network model, shorten the model inference time, and reduce the model power consumption. Summary of the invention

[0006] The present invention aims to provide a method for optimizing the reasoning performance of a neural network model, an artificial intelligence chip, and a computing device, which can optimize the loading process of the neural network model, shorten the model reasoning time, and reduce the model power consumption.

[0007] According to one aspect of the present invention, a method for optimizing the inference performance of a neural network model is provided, the method comprising:

[0008] Pre-running the directed graph structure of the neural network model includes: sequentially acquiring each operator of each layer from the directed graph structure; determining the type of the operator; and classifying the operator according to the type of the operator;

[0009] According to the result of the classification processing, the processed operator is added to the operator execution queue;

[0010] Get input data:

[0011] Operators are scheduled in sequence from the operator execution queue to perform inference calculations on the input data.

[0012] According to some embodiments, classifying the operators according to the types of the operators includes:

[0013] If the operator is a constant operator, directly allocate operating system memory to write the constant value.

[0014] According to some embodiments, classifying the operators according to the types of the operators includes:

[0015] If the operator is not a constant operator, further determining whether the operator is a dynamic type;

[0016] If the operator is of a dynamic type, further determining whether the operator can perform shape derivation;

[0017] If the operator is of a dynamic type but cannot perform shape deduction, the operator is directly added to the operator execution queue.

[0018] According to some embodiments, classifying the operators according to the types of the operators includes:

[0019] If the operator is of a non-dynamic type, or is of a dynamic type and can perform shape deduction, the memory size required for the output of the operator is calculated according to the output dimension value and data type of the operator;

[0020] Further determining whether the operator is a deformation operator;

[0021] If the operator is a deformation operator, the output memory of the operator directly reuses the input memory, and no additional memory allocation and data copying are performed.

[0022] According to some embodiments, the operator is classified and processed according to the type of the operator, further comprising: if it is further determined that the operator is not a deformation operator, the address offset value of the operator in a continuous memory of a static memory pool is calculated according to the memory size required for the output, and the value of the flag bit of the operator is set to a static data type, so that the required memory block is obtained according to the value of the flag bit and the address offset value during actual operation;

[0023] Performing parameter calculation on the operator;

[0024] Add the segment to the operator execution queue.

[0025] According to some embodiments, scheduling operators from the operator execution queue to perform inference calculations on the input data in sequence includes:

[0026] If the operator is a hardware operator, the relevant parameters are packaged and a request is sent to the artificial intelligence computing core so that the hardware operator is executed. Otherwise, it is scheduled to the central processing unit of the processing core for execution, and the execution result is written into the dynamic random access memory output by the operator.

[0027] According to some embodiments, scheduling operators from the operator execution queue to perform inference calculations on the input data in sequence further includes:

[0028] Allocate a contiguous section of DRAM memory;

[0029] The dynamic random access memory is partitioned and managed, and the dynamic random access memory is divided into a static memory area and a dynamic memory area, wherein the static memory area includes a model parameter memory, an operator static total memory, and a hardware operator intermediate result memory, wherein:

[0030] The model parameter total memory is used to store model parameters, and the parameter values ​​required by the operator are directly obtained according to the address and size of the model parameters during inference calculation;

[0031] The operator static total memory is used to store the total memory required for all static operators and dynamic operator outputs capable of shape derivation, and the total memory is the total memory size managed by the static memory pool calculated during the pre-run processing;

[0032] The total memory of intermediate results of hardware operators is used to store the intermediate result data of the hardware operators in the calculation process after the hardware operators are divided into blocks. The size of the total memory of intermediate results of hardware operators is obtained in the pre-run processing;

[0033] The dynamic memory area is managed by a dynamic memory pool, and the output memory of the dynamic operator that cannot perform shape deduction is allocated from the dynamic memory pool of the dynamic memory area during the inference calculation process.

[0034] According to some embodiments, the method further comprises continuously managing the static memory pool, wherein:

[0035] When allocating memory, the free memory hash table is used to search and allocate the memory block size required by the operator, and the used memory hash table is used to count the allocated memory blocks, wherein the key value of the used hash table is the address offset value of the allocated memory block, and the search value of the used memory hash table is the allocated memory block size.

[0036] When releasing the allocated memory block, the memory block is removed from the used memory hash table according to the address offset value, the memory block is recorded in the free memory block hash table, and adjacent free memory blocks are merged.

[0037] According to some embodiments, the method further comprises: converting the neural network model file into a runtime model structure file, wherein the runtime model structure file comprises the directed graph structure and model parameters. During runtime, the directed graph structure and the model parameters are sequentially loaded in blocks.

[0038] According to another aspect of the present invention, there is provided an artificial intelligence chip, which implements any of the methods described above when performing neural network model inference.

[0039] According to another aspect of the present invention, there is provided a computing device, comprising:

[0040] Processor; and

[0041] A memory storing a computer program, which, when executed by the processor, enables the processor to execute any of the methods described above.

[0042] According to another aspect of the present invention, there is provided a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, and when the instructions are executed by a processor, the processor is caused to execute any of the above methods.

[0043] According to an embodiment of the present invention, by pre-running the loaded directed graph structure, different types of operators are distinguished and execution strategies are optimized for different operator types, thereby reducing the time consumption during the actual reasoning process, improving the reasoning efficiency of the neural network model and reducing power consumption.

[0044] According to some embodiments, by sequentially loading the directed graph structure and the model parameters in blocks, loading errors caused by memory limitations are resolved, and block loading can be adjusted according to actual memory conditions, thereby improving the flexibility of the system. For model reasoning tasks in resource-constrained environments such as embedded systems or mobile devices, it is possible to effectively overcome memory limitations, avoid loading errors, reduce energy consumption, and increase reasoning speed.

[0045] According to some embodiments, for operators that only change the dimension but not the value, since only the dimension of the operator is changed and the data size does not change, its output memory can directly reuse the input memory, and no additional memory allocation and data copying are required, thereby reducing the overhead of reallocating memory and copying values.

[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below.

[0048] Figure 1 A schematic diagram of the traditional neural network model reasoning execution process is shown.

[0049] Figure 2 A flow chart of a method for optimizing the inference performance of a neural network model according to an example embodiment is shown.

[0050] Figure 3 A schematic diagram of a method execution flow for optimizing the inference performance of a neural network model according to an example embodiment is shown.

[0051] Figure 4 A schematic diagram of the structure of a model file at runtime in a method for optimizing the inference performance of a neural network model according to an example embodiment is shown.

[0052] Figure 5 A schematic diagram of a pre-run process for optimizing the inference performance of a neural network model according to an example embodiment is shown.

[0053] Figure 6 A schematic diagram of operator engine scheduling and execution in a method for optimizing neural network model inference performance according to an example embodiment is shown.

[0054] Figure 7 A schematic diagram of runtime memory layout in a method for optimizing neural network model inference performance according to an example embodiment is shown.

[0055] Figure 8 A schematic diagram showing a continuous management of a static memory pool in a method for optimizing the inference performance of a neural network model according to an example embodiment.

[0056] Fig. 9 A schematic diagram showing a flow of allocating memory from a static memory pool in a method for optimizing neural network model inference performance according to an example embodiment.

[0057] Fig.10 A schematic diagram showing a flow of releasing memory from a static memory pool in a method for optimizing neural network model inference performance according to an example embodiment.

[0058] Fig.11 A block diagram of a computing device is shown according to an exemplary embodiment. DETAILED DESCRIPTION

[0059] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0060] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0061] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0062] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0063] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the present inventive concept. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0064] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0065] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of example embodiments, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present invention, and therefore cannot be used to limit the scope of protection of the present invention. With the rapid development of deep learning, the application of neural networks in industry has made significant progress, but the use of neural network models with many layers on limited embedded products is facing some challenges.

[0066] Since deep neural networks require a lot of computing resources, the more layers there are in the network, the longer it takes to reason about the model. In addition, generally speaking, the more complex the neural network model is, the more power it consumes during reasoning. When performing artificial intelligence reasoning, the traditional neural network model first starts the runtime engine, loads the model file from the model file, and parses the executed graph structure and related parameter values, and then performs reasoning to obtain the final result.

[0067] In traditional neural network model deployment solutions, large neural network models generally have loading problems and often fail to load. On limited embedded products, the entire model file must be loaded at once, and the model file is large. This often leads to insufficient memory due to memory limitations of the operating system (OS), causing problems with model loading. In addition, deep large neural network models often take a long time to reason, resulting in high power consumption.

[0068] To this end, the present invention proposes a method for optimizing the reasoning performance of a neural network model, an artificial intelligence chip, and a computing device, which can optimize the loading process of the neural network model, shorten the model reasoning time, and reduce the model power consumption. According to an embodiment, by pre-running the loaded directed graph structure, different types of operators are distinguished and the execution strategy is optimized for different operator types, the time consumption in the actual reasoning process is reduced, the reasoning efficiency of the neural network model is improved, and the power consumption is reduced.

[0069] Before describing the embodiments of the present invention, some terms or concepts involved in the embodiments of the present invention are explained.

[0070] TCM (Tightly Coupled Memory): Tightly coupled memory, commonly used in embedded systems. It is tightly integrated with the processor core and provides fast access speed. Due to its physical location close to the processor, TCM provides extremely low access latency. TCM can support high data transfer rates and is suitable for applications with high performance requirements. TCM is often used to store real-time data or mission-critical code, such as real-time operating systems (RTOS) and high-performance computing.

[0071] Mailbox: A mechanism for inter-process communication that allows different processing units (such as CPU, DSP or GPU) to exchange messages or data. In this mechanism, two or more entities (such as CPU and I / O devices, or two different processor cores) share a set of registers or memory areas, which are called "mailboxes". These mailboxes can be used to store messages or status information so that entities can notify each other or pass simple control information.

[0072] Static memory pool: used to manage a large continuous memory (this memory does not need to be allocated from the device), which is a virtual memory of infinite size. If an operator requests to allocate static memory, the offset value from the first address in this continuous memory area of ​​infinite size is calculated. The following is an example embodiment of the present invention described in conjunction with the accompanying drawings.

[0073] Figure 2 A flow chart of a method for optimizing the inference performance of a neural network model according to an example embodiment is shown.

[0074] According to an example embodiment, by pre-running the loaded directed graph structure, the time consumption in the actual reasoning operation process can be reduced.

[0075] See also Figure 2 , in S101, each operator of each layer is obtained in sequence from the directed graph structure.

[0076] According to some embodiments, a directed graph structure is read from a runtime model structure file, the structure including information such as the execution order of operators, operator objects, and input / output objects of operators, etc. Each layer in the directed graph structure is traversed in turn, and each operator in the layer is obtained.

[0077] In S103, the type of the operator is determined, and the types of the operator include: a constant operator (Const), a deformation operator (reshape / reshape_like), a hardware operator, and a user-defined operator other than the above three operators.

[0078] According to some embodiments, for each acquired operator, its type is determined. Operator types include, but are not limited to: constant operator (Const), deformation operator (reshape / reshape_like), hardware operator, and custom operators other than the above three operators. Among them, the output of the constant operator (Const) is a constant value and does not depend on the input data. The deformation operator (reshape / reshape_like) is used to change the shape or size of the input data. The hardware operator is an operator directly executed by hardware (such as FPGA, ASIC, etc.). The rest are custom operators.

[0079] In S105, the operators are classified according to their types.

[0080] According to some embodiments, the operators are classified and pre-run according to the types of the operators, and the required memory size is allocated according to the actual operation requirements of different operators to save inference time and power consumption during the actual operation.

[0081] In S107, according to the result of the classification processing, the processed operator is added to the operator execution queue.

[0082] According to some embodiments, the operators are classified and pre-run, and after the required memory size is allocated according to the actual running requirements of different operators, the processed operators are added to the operator execution queue according to the type.

[0083] In S109, input data is acquired.

[0084] After completing the above pre-run processing, you can obtain data from the dataset and start directed graph reasoning, that is, directed graph reasoning, which is actually the process of executing each layer of operators in the directed graph.

[0085] In S111, operators are scheduled in sequence from the operator execution queue to perform inference calculations on the input data.

[0086] According to some embodiments, after the input data is prepared, each layer of operators in the directed graph is scheduled to be executed on the inference engine. Each operator is scheduled to be executed on the inference engine in turn according to the order in the operator execution queue. The operator is executed on the inference engine to obtain the result of the layer. During scheduling, it is ensured that the input data and parameters required by the operator are ready.

[0087] Figure 3 A schematic diagram of a method execution flow for optimizing the inference performance of a neural network model according to an example embodiment is shown.

[0088] See also Figure 3, the figure shows a schematic diagram of the execution flow of a method diagram for optimizing the reasoning performance of a neural network model according to an example embodiment, wherein the original neural network model is converted into a runtime model file. When the runtime is started, the directed graph structure in the runtime model file is loaded into the graph object in blocks, and the model parameters are loaded into the device dynamic random access memory (DDR) memory in blocks. The loaded directed graph object is pre-processed. A frame of data is obtained from the image data set and reasoning is started. Each layer of operators in the directed graph is scheduled to be executed on the corresponding execution core to obtain the reasoning result.

[0089] Figure 4 A schematic diagram of the structure of a model file at runtime in a method for optimizing the inference performance of a neural network model according to an example embodiment is shown.

[0090] See also Figure 4 , the figure shows a schematic diagram of the runtime model file structure in the method for optimizing the reasoning performance of a neural network model according to an example embodiment. The model file trained by the neural network is converted into a runtime model file, and the runtime model file includes two parts. One part is a directed graph structure, which includes the execution order of the operator, the operator object, and the input / output object of the operator, etc. The second part is the model parameter, which includes the operator sequence number and the weight, bias and other parameter values ​​required by the operator. After the directed graph structure and the model parameters are loaded in blocks in sequence, at runtime, according to the execution sequence number of the operator, a certain operator object and its corresponding input / output, as well as the related parameters required by the operator are obtained, and finally dispatched to the engine to execute to obtain the result of the layer.

[0091] Figure 5 A schematic diagram of a pre-run process for optimizing the inference performance of a neural network model according to an example embodiment is shown.

[0092] See also Figure 5 , the figure shows a schematic diagram of a pre-run process for optimizing the inference performance of a neural network model according to an example embodiment.

[0093] At s1, each operator is obtained from the directed graph structure in the order in which the operators are executed.

[0094] In s2, the operators are classified and processed according to their types. If the operator meets the hardware operator conversion condition, the operator is converted into a hardware operator. If the operator is a constant operator, the operating system memory is directly allocated to write the constant value.

[0095] According to some embodiments, the hardware operator can directly obtain data on the DDR through DMA by configuring the register address, and can also use faster RAM and SCRACHPAD as storage media to store the intermediate results of the operator. Converting the software operator to a hardware operator can speed up the speed at which the engine obtains data from the storage medium. The parameters related to the hardware operator are configured through registers to increase the data acquisition speed. For example, the execution command (ADD, MUL and other instructions) is directly configured in the register to speed up the execution of the operator. Compared with not converting to a hardware operator, this method greatly reduces the time consumption, can improve the model execution efficiency, and shorten the running time.

[0096] According to some embodiments, the purpose of converting to a hardware operator is to make full use of the storage, computing, accelerator resources, instruction sets and registers of the hardware product to accelerate the execution of the operator. The conditions met include: feasibility, determinism, repeatability, computational complexity, and resource utilization, that is, the operator has a certain parallel computing capability. Hardware implementations are usually able to process multiple data streams at the same time. The behavior of the operator should be deterministic, that is, it always produces the same output under the same input. This helps to achieve consistency and predictability in hardware. If the operator needs to be repeatedly executed on multiple data sets or is frequently called in a streaming data processing scenario, it will be more efficient to convert it to a hardware operator. The computational complexity of the operator should be moderate. Overly complex operators may be difficult to implement in hardware or may cause performance bottlenecks. The operator should be able to effectively utilize hardware resources (such as memory, computing units, etc.). If the resource consumption of the operator is too high, it may lead to uneconomical hardware implementation.

[0097] According to some embodiments, a type of operator that cannot be hardwareized (except constant operators and deformation operators) is constructed as a custom operator, and this type of operator is usually scheduled for execution on an ARM CPU core of an embedded device product.

[0098] In s3, if the operator is not a constant operator, it is further determined whether the operator is of a dynamic type.

[0099] If the operator is of a dynamic type, it is further determined whether the operator can perform shape deduction (shape deduction).

[0100] If the operator is of a dynamic type but cannot perform shape deduction, the operator is directly added to the operator execution queue.

[0101] For example, determine whether reshape / reshape_like, custom operators, and hardware operators are dynamic types. Operators with dynamic properties require shape derivation during pre-run or execution. The output shape of the operator cannot determine a certain dimension value during the graph construction process. This is called an operator with dynamic properties. The output shape value of the operator needs to be determined in the pre-run based on all the input shape values ​​of the operator and some parameters of the operator. During the process of building a directed graph, the value of a certain dimension in the operator output shape [n, h, w, c] cannot be determined. For those that cannot perform shape derivation, they are directly added to the s7 operator execution queue.

[0102] According to some embodiments, if the operator is of a non-dynamic type, or is of a dynamic type and can perform shape derivation, the memory size required for the output of the operator is calculated according to the output dimension and data type of the operator. For example, for a dynamic operator that meets the requirements of shape derivation, the output dimension value of the operator is derived according to the input and parameters of the operator, and then the data size of the operator is calculated according to the output dimension and data type of the operator.

[0103] In s4, it is further determined whether the operator is a deformation operator; if the operator is a deformation operator, the output memory of the operator directly reuses the input memory, and no additional memory allocation and data copying are performed.

[0104] According to some embodiments, it is determined whether it is an operator such as reshape / reshape_like that only changes the dimension but not the value. If it is such an operator, since only the dimension of the operator is changed and the data size does not change, its output memory directly reuses the input memory, and no additional memory is allocated or data is copied, thereby reducing the overhead of reallocating memory and copying values.

[0105] In s5, if it is further determined that the operator is not a deformation operator, the address offset value of the operator in a continuous memory of the static memory pool is calculated according to the memory size required for the output, and the value of the flag of the operator is set to a static data type, so that the required memory block is obtained according to the value of the flag and the address offset during actual operation; parameters of the operator are calculated; and the operator is added to the operator execution queue.

[0106] For example, if it is a static type operator or a custom or hardware operator whose dimensions can be derived, the address offset value of the operator in a continuous memory is calculated in the static memory pool according to the required memory size, and the operator flag is set to be of static data type. In this way, during the actual operation, the required memory block is obtained according to the flag value and the memory offset, without the need for time-consuming operations such as requesting memory allocation from the device.

[0107] In s6, the operator parameters are calculated. These parameter values ​​or flag values ​​are needed when the operator is actually executed. Among these parameters, invalidation and flushing are related to the data consistency problem caused by whether the device has high-speed memory (cache) enabled. Enabling cache can speed up the performance of the core to obtain data, but data synchronization is required. If the output of the current operator is reused by the next hardware operator, the output data of the operator needs to be flushed from the cache to the DDR physical storage. Only hardware operators can directly read / write data in DDR through direct memory access (DMA). Other types of operators need to read / write data in DDR and need to map the DDR memory cache to the virtual address space before reading and writing. In order to obtain data faster, the central processing unit (CPU) will obtain it from the cache. If a cache miss occurs, the data in the virtual address space mapped by DDR is read into the cache so that the CPU can obtain the data. Usually, in order to maintain data consistency, it is necessary to frequently refresh cache data to the virtual address space memory mapped by DDR memory. If it is a hardware operator, it is directly obtained from DDR for execution.

[0108] According to some embodiments, since the operator input and output are stored in the DDR physical memory, if it is a custom software operator, it needs to be executed based on the reduced instruction set computing (RISC) architecture (ARM) core CPU. The CPU accesses the virtual address space and needs to map the physical address space to the virtual address space. The output of a hardware operator may be given to multiple software operators. In order to speed up the reading and writing of data, when converting the DDR physical address hardware to the virtual address, the cache is enabled to reduce the reading and writing time of the data. If the output of the software operator is given to the hardware operator, the software operator output needs to be flushed so that the data read by the hardware operator is complete. If the output of the hardware operator is given to the software operator, the hardware operator needs to be invalidated / flushed so that the data read and written by the software operator is complete.

[0109] At s7, the processed operator is added to the operator execution queue.

[0110] Such operations reduce the number of constant operators and static deformation operators in the execution process, so that the execution of these operators does not need to rely on input data, but can be executed in the pre-run process in advance, reducing the time consumed by the neural network model to execute and schedule operators during the reasoning process and the power consumed by the operation, improving the model reasoning efficiency and reducing the overall operating power consumption.

[0111] Figure 6 A schematic diagram of operator engine scheduling and execution in a method for optimizing neural network model inference performance according to an example embodiment is shown.

[0112] See also Figure 6 , operators are scheduled from the operator execution queue in sequence to perform inference calculations on the input data. If the operator is a hardware operator, the relevant parameters are packaged and a request is sent to the artificial intelligence computing core so that the hardware operator is executed. Otherwise, it is scheduled to the central processing unit of the processing core for execution, and the execution result is written into the dynamic random access memory output by the operator.

[0113] According to some embodiments, a frame of data is obtained from the data set, and directed graph reasoning is started. If it is a software operator, it is scheduled to be executed on the artificial intelligence chip (AI CPU), and the execution result is written to the DDR memory output by the operator. The DDR memory is the memory mapped to the virtual address space with the cache opened in advance. Only in this way can the CPU core access this data. Constant operators are allocated memory in the pre-run, and constant operators are not included in the execution queue. If the input of other operators is the output of the constant operator, the value can be directly obtained through the address.

[0114] According to some embodiments, for a hardware operator, if the input of the hardware operator reuses the output DDR memory of the previous operator, and the result of the previous operator is written into the DDR memory after execution, the hardware operator packages the related parameters and requests the AI ​​core to execute the hardware operator. Figure 6, the hardware operator packages the parameters and writes the data batches into the TCM cache. When the data is written, the mailbox interface is called to send a request execution signal. When the AI ​​core receives the signal, it reads the relevant parameters of the hardware operator from the TCM and stores the input data in the DDR memory in advance. In this way, the data required for the hardware operator execution is ready and waiting for the engine to schedule it for execution. After the engine scheduler on the AI ​​core receives the hardware operator inference request, it queries the available AI core resources and schedules it for execution. Among them, the input data is directly obtained from the DDR through DMA. After execution, the hardware operator is written to the DDR memory output by the operator through DMA, and the information of execution success or failure is packaged and written to the TCM. The execution completion signal is sent through the mailbox, so that the execution process of the entire hardware operator is completed.

[0115] According to some embodiments, after the model is converted into a runtime model file, the directed graph contains only 4 types of operators: Const (constant operator), reshape / reshape_like deformation operator, custom operator, and hardware operator. Among them, hardware operators refer to specific computing functions implemented at the computer hardware level. These operators are usually used to process basic mathematical operations, such as addition, subtraction, multiplication and division, or more complex operations, such as matrix operations, convolution operations, etc. The hardware operator is implemented by a dedicated circuit and can be executed in parallel inside the processor, thereby improving computing efficiency. The software operator is hardwareized into a hardware operator, and some two-dimensional convolution (Conv2D), pooling (Pool), and activation function (ReLu) can be converted into hardware operators through addition, subtraction, multiplication, division, and convolution. In view of the current embedded acceleration product features on the market, the hardware acceleration engine is Tensor and Vector, so the hardware operator is divided into two types of operators, Te and Ve, for this acceleration feature. The execution design of the hardware operator includes writing input to DDR, packaging parameters and writing to TCM, sending execution requests to hardware, hardware scheduling execution, writing results to DDR after execution, writing status information to TCM, and sending back a response signal of execution completion. Each hardware operator needs to be processed according to this processing link, resulting in too long runtime for actual operation. To address this problem, multiple consecutive hardware operators of the same type (Te or Ve) are merged into a large package and requested as a whole. In order to accelerate the execution of large packages on the AI ​​core, the large package is divided into n blocks, and the storage location of each block in the hardware operator is set in advance. If the block output meets the storage size requirements of high-speed access storage areas (RAM), temporary storage areas (SCRCAHPAD), etc., which are only 4M in size, the block output is set to these high-speed storage areas instead of DDR with relatively slow access speed, which can reduce the overall execution time of the operator. Custom operators are such operators implemented in C / C++ based on the principles of the algorithm.

[0116] Figure 7 A schematic diagram of runtime memory layout in a method for optimizing neural network model inference performance according to an example embodiment is shown.

[0117] See also Figure 7, when the operators are sequentially scheduled from the operator execution queue to perform inference calculation on the input data, a continuous dynamic random access memory memory can be allocated; the dynamic random access memory memory is partitioned and managed, and the dynamic random access memory memory is divided into a static memory area and a dynamic memory area, wherein the static memory area includes a total model parameter memory, an operator static total memory, and a hardware operator intermediate result total memory, wherein the model parameter total memory is used to store model parameters, and the parameter values ​​required by the operator are directly obtained according to the address and size of the model parameters during inference calculation; the operator static total memory is used to store the total memory required for all static operators and dynamic operator outputs that can perform shape deduction, and the total memory is the total memory size managed by the static memory pool calculated during the pre-run processing; the hardware operator intermediate result total memory is used to store the intermediate result data of the hardware operator during the calculation process after the hardware operator is divided into blocks, and the size of the hardware operator intermediate result total memory is obtained in the pre-run processing; the dynamic memory area is managed by a dynamic memory pool, and the output memory of the dynamic operator that cannot perform shape deduction is allocated from the dynamic memory pool of the dynamic memory area during the inference calculation process.

[0118] According to some embodiments, see Figure 7 After the directed graph structure is loaded, the directed graph in the runtime model file is pre-run. During the pre-run, the static type operator or the dynamic operator that can deduce the shape value will be given a static memory pool to calculate the starting offset value of the memory. After the pre-run is completed, the memory block size required by all operators that need to allocate memory is counted.

[0119] In the process of executing the directed graph structure, first, a continuous DDR memory is requested to be allocated from the device, and the dynamic random access memory is partitioned and managed, and the dynamic random access memory is divided into a static memory area and a dynamic memory area. The static memory area includes the total memory of model parameters, the total static memory of operators, and the total memory of hardware operator intermediate results. The size of the static memory area is the total memory size required by the static memory pool during the pre-run process.

[0120] According to some embodiments, the model parameter total memory is used to store model parameters, and the parameter values ​​required by the operator are directly obtained according to the address and size of the model parameters during inference calculation; the operator static total memory is used to store the total memory required for all static operators and dynamic operator outputs that can perform shape deduction, and the total memory is the total memory size managed by the static memory pool calculated during the pre-run processing; the hardware operator intermediate result total memory is used to store the intermediate result data of the hardware operator during the calculation process after blocking, and the size of the hardware operator intermediate result total memory is obtained in the pre-run processing.

[0121] According to some embodiments, the dynamic memory area is managed by a dynamic memory pool, and the output memory of the dynamic operator that cannot perform shape derivation is allocated from the dynamic memory pool of the dynamic memory area during the inference calculation process. Automatic expansion processing is supported when the memory is insufficient, which can better adapt to dynamic shape derivation.

[0122] Figure 8 A schematic diagram showing continuous memory management for a static memory pool in a method for optimizing neural network model inference performance according to an example embodiment.

[0123] See also Figure 8 In the figure, b0~b3 represent four consecutive different memory blocks, among which b0 and b2 blocks have been allocated, and b1 and b3 are still idle. If the next block needs to allocate memory, it will be found from the free memory blocks of b1 and b3 to see if it meets the minimum demand requirements. If none of them meet the requirements, it will be allocated from the back of b3 block.

[0124] According to some embodiments, in actual operation, the static memory pool does not need to allocate a piece of continuous memory from the device DDR for management. It only virtualizes a piece of memory of infinite size. If an operator requests to allocate static memory, the offset value from the first address in this continuous memory area of ​​infinite size is calculated. After the entire directed graph structure is pre-run, the total memory size required will be counted. Afterwards, during the actual execution of the directed graph structure, a continuous memory area is divided from the continuous DDR memory allocated on the device as the static memory area. In this way, the memory allocated during the pre-run process can be calculated by the method of static memory area first address + address offset value to obtain the starting address of the memory.

[0125] Fig. 9 A schematic diagram of a process of allocating memory from a static memory pool in a method for optimizing neural network model inference performance according to an example embodiment is shown.

[0126] Fig.10 A schematic diagram showing a flow of releasing memory from a static memory pool in a method for optimizing neural network model inference performance according to an example embodiment.

[0127] See also Fig. 9 , the figure shows a schematic diagram of a process of allocating memory from a static memory pool in a method for optimizing the inference performance of a neural network model according to an example embodiment, wherein the method further includes continuously managing the static memory pool, including: when allocating memory, using a free memory hash table to search and allocate the memory block size required by the operator, and using a used memory hash table to count the allocated memory blocks, wherein the key value of the used memory hash table is the address offset value of the allocated memory, and the retrieval value is the allocated memory block size.

[0128] See also Fig. 9 , assuming that the memory size requested by the user is m0, byte alignment is performed to 128 bytes, and the minimum memory requirement actually required for allocation is m1. In the free memory hash table, check whether there is a free memory block that meets the minimum memory requirement. Assume that if a free block that meets the requirement is bi, remove the memory block bi from the free hash table. If the free memory block hash table is empty, allocate from the back of the allocated continuous memory block. If the last block in the free memory block hash table is the last block in the continuous memory area, directly return the address of the last block to the memory allocation request. For example, Fig.10 In the example, b3 is the free memory block, and it is the last block in the continuous memory. If a request is made to allocate memory, since static memory manages a large continuous memory, the starting address of the allocated block is the starting address of the b3 free block. After that, the used memory hash table (used_map) is used to count the allocated memory blocks, where the key value (key) is the address offset value of the allocated memory, and the search value (value) is the size of the allocated memory block.

[0129] According to some embodiments, in the process of searching for allocated memory, if the difference between the size of the free bi memory block and the size of the m1 to be allocated is greater than the T1 threshold, the free block bi can be split and the split block bj is added to the free memory hash table. Such a mechanism can cut the free block when it is too large, further optimize the memory space, and improve memory utilization.

[0130] See also Fig.10 , the figure shows a schematic diagram of a process of releasing memory from a static memory pool according to an example method for optimizing the inference performance of a neural network model. The method further includes continuously managing the static memory pool, wherein: when releasing an allocated memory block, according to the address offset value, removing the memory block from the used memory hash table, recording the memory block in the free memory block hash table, and merging adjacent free memory blocks.

[0131] According to some embodiments, assuming that a user requests to release a memory block bk, the address offset value of the memory block is used to remove it from the used memory hash table (used_map), and the bk block is inserted into the free memory block hash table (free_map). Determine whether the next memory block adjacent to the current memory block, with a starting address of: bk_offset + bk_size, is a free block. If so, the current memory block bk and the next memory block bk+1 need to be merged into a large block bk, where the starting address of the memory block bk is bk_offset and the memory size is the memory size of bk_size + bk+1. Afterwards, update the bk block information in the free memory block hash table. If the previous memory bk-1 of the current block bk is also free, then merge bk and bk-1 into a larger free memory block with a starting address of bk-1_offset and a size of bk-1_size + bk_size. Update the memory information of bk-1 in the free memory block hash table and remove the bk block before the merge from the free memory block hash table.

[0132] According to some embodiments, assuming that the memory distribution of the static memory pool at a certain moment is as described above Figure 8 As shown, that is, b1 and b3 memory blocks are stored in the free memory block hash table, and b0 and b2 memory blocks are allocated by the user and stored in the used memory hash table (used_map). If the memory block b0 is released, since b0 does not have a previous memory block, and the next memory block b1 of bo is free, then b0 and b1 memory blocks are merged into a large memory block b0 of size bo_size + b1_size, the b0 memory block information is removed from used_map, the b1 free block is removed from free_map, and the merged b0 large memory block is inserted into the free memory block hash table for zone management. If the released memory block is b2, since the next memory block of b2 is b3, which is free, b2 and b3 are merged, and the starting address after the merger is the starting address of the b2 memory block, and the size is b2_size + b3_size. Since the previous memory block b1 of b2 is also a free memory block, b1 and the memory block of b2&b3 are merged into a large memory block with the starting address of the b1 memory block and the size of b1_size + b2_size + b3_size. b2 is removed from used_map, b3 is removed from free_map, and the b1 memory block information in the free memory block hash table is updated.

[0133] According to some embodiments, the method further comprises: converting the neural network model file into a runtime model structure file, wherein the runtime model structure file comprises the directed graph structure and model parameters. During runtime, the directed graph structure and the model parameters are sequentially loaded in blocks.

[0134] According to some embodiments, the trained neural network model file is converted into a runtime model structure file. The converted runtime model structure file includes two parts: a directed graph structure and model parameters, wherein the directed graph structure includes information such as the execution order of operators, operator objects, and input / output objects of operators; and the model parameters include operator numbers and parameter values ​​such as weights and biases required by operators.

[0135] According to some embodiments, in order to avoid loading failures due to large models, the solution of the present invention adopts a block loading method. The directed graph structure is divided into several blocks and loaded into the memory block by block. The loading is carried out according to a certain logic or strategy (such as by layer, by functional module, etc.). Similarly, the model parameters are also divided into blocks. At runtime, as the directed graph structure is loaded, the corresponding model parameters are gradually loaded. By loading the directed graph structure and the model parameters in blocks in sequence, loading errors caused by memory limitations are solved, and the block loading can be adjusted according to the actual memory situation, thereby improving the flexibility of the system.

[0136] According to some embodiments, since some deep and large models contain many operators and parameters, if they are loaded all at once, the loading will fail due to limited OS memory. This problem can be solved by converting the model file and loading the converted directed graph structure and model parameters in blocks, and loading them block by block during operation.

[0137] According to some embodiments, the problem of memory limitation is solved by loading in blocks. Block loading can be adjusted according to the actual memory situation, which improves the flexibility of the system. By pre-running processing and distinguishing different types of operators, the execution strategy can be optimized in a targeted manner, reducing the calculation time during the reasoning process. For model reasoning tasks in resource-constrained environments such as embedded systems or mobile devices, it can effectively overcome memory limitations and improve reasoning speed.

[0138] According to some embodiments, the design scheme of the present invention is based on the characteristics of embedded products on the current market. In view of the problem that some large neural network models are loaded at one time on the limited OS memory and will fail to load due to insufficient OS memory, it is proposed to convert the trained original model into a runtime model file through the model, separate the directed graph structure and parameter model in the model, and load all models in batches in sequence. After loading the runtime directed graph structure, the directed graph structure in the model is pre-run, that is, before the inference graph, the static memory pool is used to allocate memory for all types of operators in the graph and dynamic operators that can derive shape values, that is, to determine the address offset value of the operator in the memory relative to the first address. Some time-consuming calculations in the reasoning process are advanced to the pre-run, for example, the relevant parameter values ​​that need to be used when the operator is executed are calculated in advance, and the flag mark values ​​of which operators need to refresh the data from the cache to the DDR memory after execution are calculated. Such a method can greatly reduce the operation time of the neural network model, reduce the power consumption of the neural network model, and improve the reasoning efficiency of the neural network model.

[0139] According to some embodiments, the design scheme of the present invention uses the space-for-time approach to reduce the time-consuming DDR memory allocation and release of the input / output, parameters, etc. of the operator during reasoning of the directed graph structure, and adopts a partition management mechanism for the DDR memory allocated from the device, which is divided into two areas: the static memory area and the dynamic memory area. Among them, the static memory area is divided into three continuous small segments, including: the total memory of model parameters, the total static memory of operators, and the total memory of the intermediate results of hardware operators; the dynamic memory area uses a dynamic memory pool to manage the DDR memory allocated from the device, and the dynamic memory area supports an automatic expansion mechanism, so that some dynamic type operators that cannot derive dimension values ​​during the pre-run process can have sufficient address space to store the intermediate result data in the dynamic derivation process during reasoning execution, and the storage management is more flexible and adaptable to a wider range of actual application scenarios and simulation scenarios, reducing loading errors and derivation errors caused by memory limitations, and further optimizing the reasoning performance of the neural network model.

[0140] According to some embodiments, the execution of some operators in the directed graph structure does not need to rely on input data or does not change the input data. In the design scheme of the present invention, the execution of this type of operator is advanced to the pre-run process. In the actual execution of reasoning, it is not necessary to execute this type of operator, and the results obtained in the pre-run process can be directly called. This method can reduce the time consuming of the entire reasoning.

[0141] According to some embodiments, in the design of the present invention, when an operator is scheduled to be executed on the engine, if the operator input / output dynamic random access memory (DDR) memory is from the operator static memory area managed by the Static memory area, the corresponding DDR memory block can be directly found through the address offset value. If the operator needs a parameter value, it is directly obtained from the total memory of the parameter model managed by the Static memory area, thereby reducing the allocation of dynamic random access memory (DDR) memory for input / output and parameters, the operation of reading parameters into the DDR memory, and the time-consuming execution of releasing the DDR memory of the operator input and parameters when executing the operator, thereby improving the model operation efficiency and greatly shortening the reasoning time of the neural network model.

[0142] Fig.11 A block diagram of a computing device according to an example embodiment of the invention is shown.

[0143] like Fig.11 As shown, computing device 30 includes processor 12 and memory 14. Computing device 30 may also include bus 22, network interface 16, and I / O interface 18. Processor 12, memory 14, network interface 16, and I / O interface 18 may communicate with each other via bus 22.

[0144] The processor 12 may include one or more general-purpose CPUs (Central Processing Units, processors), microprocessors, or application-specific integrated circuits, etc., for executing relevant program instructions. According to some embodiments, the computing device 30 may also include a high-performance graphics card (GPU) 20 for accelerating the processor 12.

[0145] The memory 14 may include a machine system readable medium in the form of a volatile memory, such as a random access memory (RAM), a read-only memory (ROM) and / or a cache memory. The memory 14 is used to store one or more programs including instructions and data. The processor 12 may read the instructions stored in the memory 14 to execute the above-mentioned method according to the embodiment of the present invention.

[0146] The computing device 30 may also communicate with one or more networks via the network interface 16. The network interface 16 may be a wireless network interface.

[0147] The bus 22 may include an address bus, a data bus, a control bus, etc. The bus 22 provides a path for exchanging information between components.

[0148] It should be noted that, in the specific implementation process, the computing device 30 may also include other components necessary for normal operation. In addition, those skilled in the art may understand that the above device may only include components necessary for implementing the embodiments of this specification, and need not include all components shown in the figure.

[0149] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when the program is executed by a processor. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), a network storage device, a cloud storage device, or any type of medium or device suitable for storing instructions and / or data.

[0150] An embodiment of the present invention also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any method recorded in the above method embodiments.

[0151] Those skilled in the art can clearly understand that the technical solution of the present invention can be implemented with the help of software and / or hardware. "Unit" and "module" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a field programmable gate array, an integrated circuit, etc.

[0152] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0153] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0154] In the several embodiments provided by the present invention, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention.

[0158] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0159] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended clauses.

Claims

1. A method for optimizing the inference performance of a neural network model, characterized in that: The method comprises: Pre-run the directed graph structure of the neural network model, including: Obtain each operator of each layer from the directed graph structure in turn, Determine the type of the operator, The operators are classified according to their types: If the operator is not a constant operator, it is further determined whether the operator is of a dynamic type. If the operator is of a non-dynamic type, or is of a dynamic type and can perform shape deduction, the memory size required for the output of the operator is calculated according to the output dimension value and data type of the operator, and it is further determined whether the operator is a deformation operator. If the operator is a deformation operator, the output memory of the operator directly reuses the input memory, and no additional memory allocation and data copying are performed; According to the result of the classification processing, the processed operator is added to the operator execution queue; Get input data; Operators are scheduled in sequence from the operator execution queue to perform inference calculations on the input data.

2. The method according to claim 1, characterized in that Classifying the operators according to the types of the operators also includes: If the operator is a constant operator, directly allocate operating system memory to write the constant value.

3. The method according to claim 2, characterized in that Classifying the operators according to the types of the operators also includes: If the operator is of a dynamic type, further determining whether the operator can perform shape derivation; If the operator is of a dynamic type but cannot perform shape deduction, the operator is directly added to the operator execution queue.

4. The method according to claim 1, characterized in that: Classifying the operators according to the types of the operators also includes: If it is further determined that the operator is not a deformation operator, then according to the memory size required for the output, the address offset value of the operator in a continuous memory of the static memory pool is calculated, and the value of the flag bit of the operator is set to a static data type, so that the required memory block is obtained according to the value of the flag bit and the address offset value during actual operation; Performing parameter calculation on the operator; Add the operator to the operator execution queue.

5. The method according to claim 1, characterized in that: The operators are sequentially dispatched from the operator execution queue to perform reasoning calculations on the input data, including: If the operator is a hardware operator, the relevant parameters are packaged and a request is sent to the artificial intelligence computing core so that the hardware operator is executed. Otherwise, it is scheduled to the central processing unit of the processing core for execution, and the execution result is written into the dynamic random access memory output by the operator.

6. The method according to claim 5, characterized in that The method further includes: scheduling operators from the operator execution queue to perform reasoning calculations on the input data in sequence; Allocate a contiguous section of DRAM memory; The dynamic random access memory is partitioned and managed, and the dynamic random access memory is divided into a static memory area and a dynamic memory area, wherein the static memory area includes a total memory of model parameters, a total static memory of operators, and a total memory of intermediate results of hardware operators, wherein: The model parameter total memory is used to store model parameters, and the parameter values ​​required by the operator are directly obtained according to the address and size of the model parameters during inference calculation; The operator static total memory is used to store the total memory required for all static operators and dynamic operator outputs capable of shape derivation. The operator static total memory is the total memory size managed by the static memory pool calculated during the pre-run process; The total memory of intermediate results of hardware operators is used to store the intermediate result data of the hardware operators in the calculation process after the hardware operators are divided into blocks. The size of the total memory of intermediate results of hardware operators is obtained in the pre-run processing; The dynamic memory area is managed by a dynamic memory pool, and the output memory of the dynamic operator that cannot perform shape deduction is allocated from the dynamic memory pool of the dynamic memory area during the inference calculation process.

7. The method according to claim 4, characterized in that The method further includes continuously managing the static memory pool, wherein: When allocating memory, the free memory hash table is used to search and allocate the memory block size required by the operator, and the used memory hash table is used to count the allocated memory blocks, wherein the key value of the used memory hash table is the address offset value of the allocated memory block, and the search value of the used memory hash table is the allocated memory block size; When releasing the allocated memory block, the memory block is removed from the used memory hash table according to the address offset value, the memory block is recorded in the free memory hash table, and adjacent free memory blocks are merged.

8. The method according to claim 1, characterized in that: The method further comprises: Converting the neural network model file into a runtime model structure file, wherein the runtime model structure file includes the directed graph structure and model parameters; During runtime, the directed graph structure and the model parameters are loaded in blocks in sequence.

9. An artificial intelligence chip, characterized in that: The artificial intelligence chip implements the method as described in any one of claims 1-8 when performing neural network model inference.

10. A computing device, characterized in that include: processor; as well as A memory storing a computer program, which, when executed by the processor, enables the processor to perform the method according to any one of claims 1 to 8.

Images