Memory management optimization method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a memory management optimization method and device, electronic equipment and a storage medium. The method comprises the following steps: allocating a continuous memory pool according to a cache required in a forward reasoning process of a neural network model; the memory pool comprises a plurality of cache blocks, and each cache block has a corresponding cache index; establishing a mapping relationship between the cache index of the cache block and the physical address of the cache block; obtaining an operator sequence obtained by compiling a forward reasoning process of the neural network model and an operation result obtained by operation of each operator in the operator sequence; allocating a cache index for each operator in the operator sequence; and determining a corresponding physical address according to the cache index of each operator and the mapping relation so as to store an operation result. According to the method, the continuous memory pool is distributed, and in the whole reasoning process, the pre-divided fixed cache blocks are multiplexed only through the index, so that generation of memory fragments is avoided, and efficient and continuous utilization of the memory is ensured.

Description

Technical Field

[0001] This invention relates to the field of memory management technology, and in particular to a memory management optimization method, a memory management optimization device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In embedded platforms, efficient memory management is crucial for ensuring system performance and stability during neural network forward inference. Existing technologies typically employ dynamic memory allocation strategies, which allocate and release caches of appropriate sizes in real-time for the intermediate feature map data required by each operator. Related technologies, by formulating a memory allocation scheme, write the data into a designated memory area within a controlled memory region. Based on key values ​​and a singly linked list, the memory space required by each kernel function during inference is obtained, and the inference result data is stored in the designated output data memory space within the controlled memory region.

[0003] However, this dynamic allocation mechanism in existing technologies is prone to memory fragmentation due to the frequent allocation and release of memory blocks of different sizes during operation. Memory fragmentation significantly reduces memory utilization and may even lead to the exhaustion of available resources due to the inability to allocate large contiguous blocks of memory, causing inference tasks to fail. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a memory management optimization method, a memory management optimization device, an electronic device, and a computer-readable storage medium that overcome or at least partially solve the above problems.

[0005] To address the above problems, a first aspect of this invention provides a memory management optimization method, the method comprising: A contiguous memory pool is allocated according to the cache required for the forward inference process of the neural network model; the memory pool includes multiple cache blocks, each with a corresponding cache index; Establish a mapping relationship between the cache index of the cache block and the physical address of the cache block; Obtain the operator sequence obtained by compiling the forward inference process of the neural network model and the operation results obtained by each operator operation in the operator sequence; Assign a cache index to each operator in the operator sequence; The corresponding physical address is determined based on the cache index of each operator and the mapping relationship to store the operation result.

[0006] Optionally, the allocation of a contiguous memory pool for the cache required by the forward inference process of the neural network model includes: Obtain the cache configuration information corresponding to the neural network model; the cache configuration information includes the required number of caches, cache type, and cache size; Based on the cache configuration information, allocate a contiguous memory pool.

[0007] Optionally, allocating a contiguous memory pool according to the cache configuration information includes: The total memory size is determined based on the cache size, and a contiguous memory pool corresponding to the total memory size is obtained; Based on the cache quantity and cache type, the contiguous memory pool is divided into multiple cache blocks, a cache index is assigned to each cache block, and the physical address of each cache block in the contiguous memory pool is determined.

[0008] Optionally, allocating a cache index for each operator in the operator sequence includes: Select a free cache block from the plurality of cache blocks as the output cache of the operator; Based on the cache index corresponding to the cache block, determine the cache index of the output cache of the operator.

[0009] Optionally, the plurality of cache blocks are divided into cache blocks for storing different types of data; the step of selecting a free cache block from the plurality of cache blocks as the output cache of the operator includes: Determine the type of output data of the operator; Select an empty buffer block from the buffer blocks corresponding to the type of the operator's output data as the operator's output buffer.

[0010] Optionally, the cache block type includes a first type of cache and a second type of cache; the first type of cache is used to store data that is not referenced by subsequent branch structure operators; the second type of cache is used to store data that will be referenced by subsequent branch structure operators. The step of selecting a free buffer block from the buffer blocks corresponding to the type of the operator's output data as the operator's output buffer includes: If the output data of the operator is of the first type, then a free cache block is selected from the first type of cache as the output cache of the operator; If the output data of the operator is of the second type, then a free cache block is selected from the second type of cache as the output cache of the operator.

[0011] Optionally, the method further includes: A cache status table is generated to record the status of each cache. After assigning an output cache index to the operator, the status of the cache is set to busy. When it is detected that the allocated cache meets the release condition, the state of the cache is set to idle; the release condition includes that the cache is not the direct input cache of the next operator and that the data stored in the cache is not referenced by other branch structure operators.

[0012] According to a second aspect of the present invention, a memory management optimization apparatus is provided, the apparatus comprising: The memory pool allocation module is used to allocate a contiguous memory pool according to the cache required for the forward inference process of the neural network model; the memory pool includes multiple cache blocks, each cache block has a corresponding cache index; The mapping relationship establishment module is used to establish the mapping relationship between the cache index of the cache block and the physical address of the cache block; The operator sequence acquisition module is used to acquire the operator sequence obtained by compiling the forward inference process of the neural network model and the operation results obtained by each operator operation in the operator sequence; A cache index allocation module is used to allocate a cache index for each operator in the operator sequence; The operation result storage module is used to determine the corresponding physical address to store the operation result based on the cache index of each operator and the mapping relationship.

[0013] Optionally, the memory pool allocation module includes: The configuration information acquisition submodule is used to acquire the cache configuration information corresponding to the neural network model; the cache configuration information includes the required number of caches, cache type, and cache size. The contiguous memory pool allocation submodule is used to allocate a contiguous memory pool according to the cache configuration information.

[0014] Optionally, the contiguous memory pool allocation submodule includes: The total memory determination unit is used to determine the total memory size based on the cache size and to obtain a continuous memory pool corresponding to the total memory size. The cache block partitioning unit is used to divide the contiguous memory pool into multiple cache blocks according to the cache quantity and cache type, assign a cache index to each cache block, and determine the physical address of each cache block in the contiguous memory pool.

[0015] Optionally, the cache index allocation module includes: The cache block selection submodule is used to select a free cache block from the plurality of cache blocks as the output cache of the operator; The cache index determination submodule is used to determine the cache index of the output cache of the operator based on the cache index corresponding to the cache block.

[0016] Optionally, the plurality of cache blocks are divided into cache blocks for storing different types of data; the cache block selection submodule includes: A data type determination unit is used to determine the type of the output data of the operator; The output buffer determination unit is used to select a free buffer block from the buffer blocks corresponding to the type of the output data of the operator as the output buffer of the operator.

[0017] Optionally, the cache block type includes a first type of cache and a second type of cache; the first type of cache is used to store data that is not referenced by subsequent branch structure operators; the second type of cache is used to store data that will be referenced by subsequent branch structure operators. The output buffer determination unit includes: The output data determination subunit is used to select a free cache block from the first type of cache as the output cache of the operator if the output data of the operator is of the first type of data; and to select a free cache block from the second type of cache as the output cache of the operator if the output data of the operator is of the second type of data.

[0018] Optionally, the device further includes: The cache status recording module is used to generate a cache status table to record the status of each cache. After assigning an output cache index to the operator, the status of the cache is set to busy. The cache state modification module is used to set the state of the cache to idle when it is detected that the allocated cache meets the release conditions; the release conditions include that the cache is not the direct input cache of the next operator and that the data stored in the cache is not referenced by other branch structure operators.

[0019] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the memory management optimization method as described in any of the preceding embodiments.

[0020] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein when executed by a processor, the computer program implements the steps of the memory management optimization method as described in any of the preceding embodiments.

[0021] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: This invention discloses a memory management optimization method, apparatus, electronic device, and storage medium. The method includes: allocating a contiguous memory pool according to the cache required for the forward inference process of a neural network model; the memory pool includes multiple cache blocks, each with a corresponding cache index; establishing a mapping relationship between the cache index of the cache block and the physical address of the cache block; obtaining the operator sequence obtained by compiling the forward inference process of the neural network model and the operation results obtained by each operator operation in the operator sequence; allocating a cache index for each operator in the operator sequence; and determining the corresponding physical address to store the operation result according to the cache index and mapping relationship of each operator. By allocating a contiguous memory pool during the initialization phase and reusing pre-divided fixed cache blocks only through index reuse throughout the inference process, memory fragmentation is avoided, ensuring efficient and continuous memory utilization, and effectively preventing inference task failure due to the inability to allocate contiguous memory. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the steps of a memory management optimization method provided in an embodiment of the present invention; Figure 2 This is a flowchart of another memory management optimization method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the neural network model structure of a memory management optimization method provided in an embodiment of the present invention; Figure 4 This is a flowchart illustrating a memory management optimization method provided in an embodiment of the present invention; Figure 5 This is a structural block diagram of a memory management optimization device provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] The current dynamic allocation mechanism is prone to memory fragmentation due to the frequent allocation and release of memory blocks of different sizes. Memory fragmentation significantly reduces memory utilization and may even lead to the exhaustion of available resources due to the inability to allocate large contiguous blocks of memory, causing inference tasks to fail.

[0025] One of the core concepts of this invention is that by allocating a contiguous memory pool during the initialization phase and reusing pre-divided fixed cache blocks only through index reuse throughout the inference process, memory fragmentation is avoided, ensuring efficient and continuous use of memory and effectively preventing inference task failure due to the inability to allocate contiguous memory.

[0026] Reference Figure 1 The diagram illustrates a flowchart of a memory management optimization method provided by an embodiment of the present invention. The method specifically includes the following steps: Step 101: Allocate a contiguous memory pool according to the cache required for the forward inference process of the neural network model; the memory pool includes multiple cache blocks, each cache block has a corresponding cache index; Existing solutions manage system memory by dynamically allocating and releasing cache. Since the cache size varies each time, repeated allocation and release of memory can lead to memory fragmentation. If memory fragmentation is not handled using sophisticated techniques, the memory may be exhausted.

[0027] In the model compilation process, this invention abstracts the neural network forward inference computation process into individual operators and pre-allocates a cache for each operator. The cache index information is recorded in the operator parameter list. Then, when performing neural network forward inference computation at the edge, it is only necessary to obtain the cache address based on the cache index, which simplifies the memory management logic at the edge and improves the speed of neural network forward inference computation at the edge.

[0028] A neural network model is a collection of mathematical computational frameworks and parameters that simulate the connection patterns of neurons in the human brain, used to learn and recognize complex patterns from data. It mainly consists of two parts: structure (architecture), which defines the skeleton of the model, i.e., how many layers it has, what type of each layer is (e.g., convolutional layer, fully connected layer), and how the layers are connected; and parameters (weights and biases), which are empirical values ​​learned by the model after training on a large amount of data and stored in the model.

[0029] The forward inference process of a neural network model refers to the complete data flow and computation sequence in which input data is fed into a pre-trained neural network model, undergoes calculations through a series of layers in the model, and finally yields the output result (such as classification probability).

[0030] A cache refers to a pre-allocated memory space used to temporarily store intermediate results during the forward inference process of a neural network. It does not store the final result or the network's weight parameters, but rather temporary data generated during the inference process.

[0031] A memory pool refers to a large, contiguous block of physical memory that is pre-allocated, once, and statically during device initialization based on the needs of the neural network model. It is a contiguous array or buffer.

[0032] A cache block refers to a contiguous memory segment with a fixed size and a unique index, pre-allocated within a statically allocated memory pool. Each cache block is used to store the output feature map (intermediate result) generated by a specific operator during the forward inference process of the neural network. Each cache block is assigned a fixed size during initialization. In this embodiment of the invention, this is mainly divided into two categories: convolution cache, used to store large temporary feature maps; and branch cache, used to store small feature maps that need to be retained.

[0033] A cache index is a simple, non-negative integer identifier that uniquely binds to a specific cache block in the memory pool. It does not contain any address information; its core function is to refer to and pass ownership and usage rights of a specific cache block between the compiler, operator stream file, and device-side executor in a way that is independent of specific memory addresses.

[0034] In this embodiment of the invention, during the compilation phase, the AI ​​compiler analyzes the complete structure of the neural network model, simulates the data flow of forward inference, and accurately analyzes the lifecycle and size of all intermediate feature maps. This determines the total number of cache blocks required, their types (such as convolutional / branch caches), and their respective sizes, and assigns a unique index to each cache block. During the device initialization phase, the edge device statically allocates a sufficient contiguous physical memory pool based on the same cache configuration determined during the compilation phase. It then divides this pool into corresponding cache blocks according to preset rules, establishing a fixed mapping relationship from "cache index" to "cache block start address," laying the foundation for efficient subsequent inference execution.

[0035] Step 102: Establish the mapping relationship between the cache index of the cache block and the physical address of the cache block; A physical address refers to the starting memory address of a cache block within a statically allocated memory pool. It is a pointer value that can be directly used by the Central Processing Unit (CPU) or the Neural Processing Unit (NPU) for data access.

[0036] In this embodiment of the invention, the mapping relationship between the cache index of the cache block and the physical address of the cache block is established during device initialization. After statically allocating a contiguous memory pool according to the predefined cache configuration (quantity, type, size), the system will create a mapping table or a pointer array programmatically.

[0037] Each predefined cache index is traversed, and the starting offset of each cache block in the memory pool is calculated and converted into an actual physical address (or a virtual address that the program can directly manipulate). This address is then permanently bound to the corresponding index. Ultimately, this mapping relationship is solidified into a resident memory lookup structure, allowing subsequent inference processes to directly and instantly find the corresponding exact physical address by simply inputting a cache index, thus achieving memory access without dynamic addressing overhead.

[0038] Step 103: Obtain the operator sequence obtained by compiling the forward inference process of the neural network model and the operation results obtained by each operator operation in the operator sequence; An operator sequence is a list of operators arranged in execution order, obtained by an AI compiler after decomposing and optimizing the forward inference process of the entire neural network model. It fully defines each computational task from input data to output result. It is an ordered list or array where each item is an "operator." Each "operator" is a specific instruction describing the specific operation to be performed, the required parameters, and the locations of the input and output data for that operation.

[0039] An operator is an abstraction and encapsulation of a basic computational operation in a neural network model. It is the smallest independently schedulable and executable computational unit that receives input data (feature maps), performs specific computations, and produces output data. Operators are divided into two categories based on their execution hardware: hardware operators, such as convolution and pooling, which can be directly accelerated by the neural network processing unit (NPU). These correspond to NPU-specific instructions in the operator stream file. Software operators, such as upsampling, are operations that the NPU does not support or is inefficient to execute and must be implemented by the central processing unit (CPU) through software code.

[0040] The result of an operator operation refers to the output feature map generated by an operator (such as convolution, pooling, etc.) after performing specific mathematical calculations based on its input data and internal parameters (such as weights). This result is a multi-dimensional array (usually a 2D or 3D tensor), which serves as the input data for subsequent operators and represents the entity of data flow and transformation in the neural network. Each operation result has a defined lifecycle, starting from when it is generated by the current operator and ending when it is no longer needed by any subsequent operators.

[0041] In this embodiment of the invention, the forward inference computation process of a neural network is abstracted into a series of operators through an AI compiler, and the operator type and operator parameters are recorded.

[0042] In this embodiment of the invention, obtaining the operator sequence refers to the process by which the device-side inference engine parses a sequentially arranged list of operators from the operator stream file loaded into memory after initialization. This sequence represents the final form of the model during the compilation phase, where each operator includes its type, parameters, and an output cache index pre-allocated statically by the compiler.

[0043] The process of obtaining the results of each operator's operation does not refer to directly obtaining the result data, but rather to triggering the computation and guiding the storage of the result. The specific process is as follows: the inference engine sequentially traverses the operator sequence. For the currently executing operator, it directly obtains the corresponding physical address based on the output cache index recorded in its parameters, through a pre-established mapping relationship. Subsequently, the engine calls the corresponding hardware (NPU) or software (CPU) computing unit to perform the operation according to the operator type, and directly writes the computation result into the aforementioned determined physical address. The entire inference process involves this step being executed repeatedly until all operators in the sequence have been executed, and the final result is stored in the output cache of the last operator.

[0044] Step 104: Assign a cache index to each operator in the operator sequence; In this embodiment of the invention, the step of allocating a cache index for each operator in the operator sequence is the core of the entire static memory management scheme. It is completed by the AI ​​compiler during the model compilation stage, and its specific process is a process of simulated execution and resource scheduling.

[0045] The compiler iterates through the parsed operator sequence and maintains a virtual cache state table to track the idle or busy state of each cache block (e.g., Index 0-4). For the current operator in the sequence, the compiler analyzes the lifecycle of its output data, focusing on whether its computation result will be referenced by subsequent branch operators such as residual joins (Shortcut) or channel concatenation (Route). Based on this assessment, the compiler allocates a suitable cache block from the "idle" cache blocks—a convolution cache if the output is temporary, and a branch cache if it needs to be retained long-term. The compiler then records the allocated cache index in the operator's parameter list and marks the cache block as "busy" in the state table.

[0046] After allocation is completed, the compiler immediately updates the cache state table and implements a cache release strategy: it checks all allocated cache blocks, and if a cache block is no longer used as the direct input of the next operator and its stored data is no longer referenced by any future operator in the sequence, it immediately resets its state to "free", thereby ensuring that the limited cache blocks can be efficiently reused throughout the inference process.

[0047] Step 105: Determine the corresponding physical address based on the cache index of each operator and the mapping relationship to store the operation result.

[0048] In this embodiment of the invention, the corresponding physical address is determined based on the cache index of each operator and the mapping relationship to store the operation result. This step occurs in real time during the inference execution phase on the device side. When an operator needs to be executed, the inference engine extracts the output cache index pre-allocated by the compiler from its parameter list. Subsequently, the engine uses this index as the key to query the "index-address" mapping table that has been established during the initialization phase. This table is usually a static array in the device configuration structure, such as dev_cfg.layer_buffer[].

[0049] Through a single efficient array indexing operation (i.e. dev_cfg.layer_buffer[2]), the engine can instantly obtain the exact starting physical address of the target cache block in the memory pool. Finally, the computing unit (NPU or CPU) directly writes the result of the operator to the memory space pointed to by that address, completing the direct storage of "result to address" without any dynamic memory management overhead.

[0050] The AI ​​compiler analyzes the neural network model, generates a sequence of operators, and statically allocates a cache index for the output of each operator, forming an operator stream file. On the device side, following the same plan, a memory pool is statically allocated, and a fixed mapping relationship between the index and physical address is established. The device executes the operators sequentially, converting the abstract index into actual physical addresses through a simple table lookup operation, thereby directly storing the computation results.

[0051] By transforming the complex and fragmented dynamic memory management problem into a compile-time one-time optimization problem and a runtime deterministic table lookup operation, high-performance, highly deterministic, and zero-memory-fragmentation neural network inference is achieved on resource-constrained edge devices.

[0052] The AI ​​compiler abstracts the forward inference computation process of the neural network into operators, records the operator type and parameters, and converts them into commands and parameter formats that the hardware can recognize. This is then encoded into a custom-formatted operator stream file, which is decoded on the chip to activate the NPU for accelerated computation. The operator stream file generated by the AI ​​compiler is divided into four parts: the first part is the header, which has a fixed length; the second part is the operator stream, a sequence of continuously executed hardware instructions implemented as the OP_HW operator; all operations that cannot be implemented in hardware must be implemented as software operators, with the execution parameters placed in the operator parameter list; the third part is the set of hardware instructions; and the fourth part is the quantized set of network weights.

[0053] The model compilation process is completed on the PC. The cache address allocated to each operator is different from the cache address allocated when performing neural network forward inference operations at the edge, but the cache index is the same. Therefore, the cache index information needs to be recorded in the operator parameter list, and then the cache address is obtained at the edge based on the cache index information.

[0054] Reference Figure 2 The diagram illustrates a flowchart of another memory management optimization method provided by an embodiment of the present invention. The method specifically includes the following steps: Step 201: Allocate a contiguous memory pool according to the cache required for the forward inference process of the neural network model; the memory pool includes multiple cache blocks, each cache block has a corresponding cache index; In this embodiment of the invention, during the compilation phase, the AI ​​compiler analyzes the complete structure of the neural network model, simulates the data flow of forward inference, and accurately analyzes the lifecycle and size of all intermediate feature maps to determine the total number, type, and size of the required cache blocks, and assigns a unique index to each cache block. During the device initialization phase, the edge device statically allocates a sufficient contiguous physical memory pool at once according to the same cache configuration determined during the compilation phase, and divides it into corresponding cache blocks according to preset rules, while establishing a fixed mapping relationship from "cache index" to "cache block start address".

[0055] In some embodiments, step 201 includes the following sub-steps: Sub-step S11: Obtain the cache configuration information corresponding to the neural network model; the cache configuration information includes the required cache quantity, cache type, and cache size; Sub-step S12: Allocate a continuous memory pool according to the cache configuration information.

[0056] The cache configuration information is the complete specification of the static memory pool required to meet the forward inference needs of a specific neural network model. This includes the number, type, size, and indexing plan of cache blocks. The total number of cache blocks defines how many independent cache blocks need to be pre-divided. The compiler analyzes the model's computation graph to find the maximum number of intermediate feature maps that need to be stored in memory simultaneously at any given time during inference. The type of each cache block categorizes the cache blocks, primarily to match their purpose and optimize memory. The first type of cache (convolution cache) is allocated to temporary feature maps with short lifecycles and large sizes. The second type of cache (branch cache) is allocated to feature maps that need to be referenced by residual connections or channel splicing layers, have long lifecycles, but may be small in size.

[0057] The size of each cache block specifies a fixed capacity for each type of cache block, determined by the sizes of the convolutional cache and the branch cache. The size of the convolutional cache is determined by the largest size of the output feature maps of all convolutional layers in the model, ensuring that it can store the most space-consuming temporary result. The size of the branch cache is determined by the largest size of all feature maps that need to be retained for the branch structure.

[0058] A memory management mechanism is added to the AI ​​compiler, pre-defining the number, size, and index of caches based on the AI ​​model structure, and statically allocating memory accordingly. The pre-defined number, size, and index of caches are based on the neural network model structure. For example, if a neural network model requires five caches, three of them temporarily store the input data needed for the channel concatenation layer, and the remaining two are double buffers, serving as input / output buffers during operator operations. The maximum cache required for convolutional layers is generally different from that required for residual connections / channel concatenation layers. To save memory, cache types are categorized into convolutional caches and branch caches.

[0059] In this embodiment of the invention, obtaining cache configuration information is a deep analysis process completed on the host side during the model compilation phase. The AI ​​compiler analyzes the complete structure of the neural network model, simulates the complete data flow of its forward inference, and accurately analyzes the lifecycle and size of all intermediate feature maps. The core of this analysis lies in determining whether the output of each operator will be referenced by subsequent branch structures such as residual connections / channel splicing layers, thereby calculating the maximum number of caches that need to be stored in memory simultaneously, and classifying the cache types accordingly (such as convolutional caches for temporary data and branch caches for retaining data). At the same time, the cache size of each type of cache is determined based on the maximum size of each type of feature map. Finally, this information is summarized into a precise cache configuration information.

[0060] Allocating a contiguous memory pool is a physical memory preparation process performed on the edge device side during device initialization. The device-side system strictly adheres to the cache configuration information obtained from the compiler (or built-in) to statically allocate a sufficient amount of contiguous physical memory in a single step. Subsequently, the system precisely divides this raw memory into predefined cache blocks according to the number, type, and size of caches defined in the configuration information, and establishes a fixed mapping relationship between the index of each cache block and its starting physical address. This completes the construction of a static memory pool dedicated to the inference of this neural network model, free from fragmentation risks.

[0061] In some embodiments, step S12 may include the following sub-steps: Sub-step S121: Determine the total memory size based on the cache size, and obtain the continuous memory pool corresponding to the total memory size; Sub-step S122: Based on the cache quantity and cache type, divide the contiguous memory pool into multiple cache blocks, assign a cache index to each cache block, and determine the physical address of each cache block in the contiguous memory pool.

[0062] In this embodiment of the invention, allocating the total memory pool is a fundamental stage in the construction of device-side memory. Based on the cache configuration information determined during the compilation phase, the system summarizes the size requirements of all cache blocks and accurately calculates the required total memory size. Subsequently, through a one-time system memory call function, a contiguous physical memory pool with a capacity exactly equal to this total size is allocated from the device memory. This step ensures the integrity and continuity of the underlying memory resources.

[0063] The system logically divides the contiguous raw memory pool according to the predefined cache quantity and cache type in the configuration information. For example, the initial part is allocated to two large convolutional caches, and the remaining part is allocated to three small branch caches. The system assigns a unique cache index to each partitioned cache block and determines its fixed starting physical address by calculating the offset of each block relative to the starting address of the memory pool, ultimately establishing an "index-address" mapping table.

[0064] By allocating a large contiguous block of memory and statically partitioning it during initialization, memory utilization and system stability are significantly improved. Static partitioning and the pre-established index-address mapping mean that the runtime only needs to look up the physical address using a simple index table, making memory access latency constant and predictable, and greatly improving inference efficiency.

[0065] Reference Figure 3 The diagram shows a schematic of a neural network model structure for a memory management optimization method provided in an embodiment of the present invention. Figure 3 This demonstrates the structure of a neural network model and the detailed configuration of its statically allocated cache memory pool. The model's operator sequence follows a computational pipeline from feature extractor to feature fusion and then to the output layer. The crucial route and upsample operators represent typical scenarios requiring branch caches to retain and pass data. The cache configuration information for this model structure and the specific implementation of the device-side memory pool initialization are shown: it explicitly defines a total of 5 cache blocks (LAYER_BUFFER_NUM5), and based on model analysis, divides them into 2 large convolutional caches (for backbone computation) and 3 small branch caches (for route layers, etc.). Finally, the required contiguous memory pool is allocated all at once using the static array ConvBuffer.

[0066] Step 202: Establish the mapping relationship between the cache index of the cache block and the physical address of the cache block; In this embodiment of the invention, the mapping relationship between the cache index of the cache block and the physical address of the cache block is established during device initialization. After statically allocating a contiguous memory pool according to the predefined cache configuration, the system creates a mapping table programmatically.

[0067] Each predefined cache index is traversed, and the starting offset of each cache block in the memory pool is calculated and converted into an actual physical address. This address is then permanently bound to the corresponding index. Ultimately, this mapping is solidified into a resident memory lookup structure, allowing for direct and instantaneous lookup of the exact physical address by simply inputting a cache index during subsequent inference, thus achieving memory access without dynamic addressing overhead.

[0068] Step 203: Obtain the operator sequence obtained by compiling the forward inference process of the neural network model and the operation results obtained by each operator operation in the operator sequence; In this embodiment of the invention, obtaining the operator sequence refers to the process by which the device-side inference engine parses a sequentially arranged list of operators from the operator stream file loaded into memory after initialization. This sequence represents the final form of the model during the compilation phase, where each operator contains its type, parameters, and an output cache index pre-allocated statically by the compiler. The inference engine sequentially traverses the operator sequence. For the currently executing operator, it directly obtains the corresponding physical address based on the output cache index recorded in its parameters, using a pre-established mapping relationship. Subsequently, the engine calls the corresponding hardware or software computing unit to perform the computation based on the operator type and directly writes the computation result into the aforementioned determined physical address.

[0069] Step 204: Assign a cache index to each operator in the operator sequence; In this embodiment of the invention, the compiler traverses the parsed operator sequence and maintains a virtual cache state table to track the idle or busy state of each cache block. For the current operator in the sequence, the compiler analyzes the lifecycle of its output data, with the core judgment being whether its calculation result will be referenced by subsequent branch operators such as residual concatenation or channel concatenation. Based on the judgment result, the compiler allocates a suitable cache block from the cache blocks with the "idle" state. If the output is temporary, a convolution cache is allocated; if it needs to be retained for a long time, a branch cache is allocated.

[0070] In some embodiments, step 204 includes the following steps: Sub-step S21: Select an empty cache block from the plurality of cache blocks as the output cache of the operator; In this embodiment of the invention, selecting a free cache block from the plurality of cache blocks as the output cache of the operator is a process performed by the AI ​​compiler during the model compilation phase. The compiler internally maintains a virtual cache state table, which records the state and type of each cache block. When traversing the operator sequence and allocating an output cache for the current operator, the compiler applies a core decision rule: determining whether the output data of the current operator will be referenced by subsequent branch structure operators. If it is determined that the data needs to be retained for later use, the compiler selects a cache block from the branch cache pool with a "free" state and allocates it to the data. If it is determined that the data is temporary, the compiler selects a cache block from the convolution cache pool with a "free" state and allocates it to the data.

[0071] Once selected, the compiler immediately marks the cache block's status as "busy" in the cache status table and records its index number in the parameter list of the current operator. This selection process is not random, but rather an optimal decision made by the compiler based on a global analysis of the entire computation graph's data flow, ensuring that data is not illegally overwritten and that memory is reused most efficiently.

[0072] In some embodiments, the plurality of cache blocks are divided into cache blocks for storing different types of data, and step S21 may include the following sub-steps: Sub-step S211: Determine the type of output data of the operator; Sub-step S212: Select an empty buffer block from the buffer blocks corresponding to the type of the output data of the operator as the output buffer of the operator.

[0073] In this embodiment of the invention, determining the type of the operator's output data is the core of the logical judgment in the allocation decision. During the compilation phase, the compiler determines the "type" of the current operator's output data by analyzing the computation graph of the neural network. Here, "type" does not refer to the data format (such as int8, fp16), but rather to its lifecycle and usage characteristics. Specifically, the core judgment criterion is whether the output data will be referenced by subsequent Shortcut or Route operators. If it is not referenced, its type is defined as temporary data. If it is referenced, its type is defined as data that needs to be retained.

[0074] Selecting a free cache block from the buffer blocks corresponding to the type of the operator's output data as the operator's output cache is a type-based resource binding. The compiler allocates from a predefined pool of corresponding cache blocks based on the type determined in S211: if the output data type is temporary, it selects from a set of free cache blocks dedicated to storing such temporary data (i.e., the convolution cache); if the output data type is data that needs to be retained, it selects from a set of free cache blocks dedicated to storing such data that needs to be retained long-term (i.e., the branch cache). By classifying cache blocks and precisely matching them with output data types of different lifecycles, it ensures that each memory resource is used for the most appropriate purpose, avoiding resource waste or allocation failures caused by allocating long-term resources for short-term data or vice versa, thereby significantly improving overall memory utilization and system reliability.

[0075] In some embodiments, the cache block type includes a first type of cache and a second type of cache; the first type of cache is used to store data not referenced by subsequent branch structure operators; the second type of cache is used to store data that will be referenced by subsequent branch structure operators, and step S212 may include the following sub-steps: Sub-step S2122: If the output data of the operator is first type data, then select an empty cache block from the first type of cache as the output cache of the operator; if the output data of the operator is second type data, then select an empty cache block from the second type of cache as the output cache of the operator.

[0076] In this embodiment of the invention, if the output data of the operator is of the first type, a free cache block is selected from the first type of cache as the output cache of the operator; if the output data of the operator is of the second type, a free cache block is selected from the second type of cache as the output cache of the operator.

[0077] "Type 1 data" specifically refers to temporary data, meaning the output of the operator will not be referenced by subsequent branching operators such as Shortcut or Route. "Type 1 cache" specifically refers to the convolution cache in this embodiment of the invention. This type of cache block is relatively large and is used for high-speed turnover of temporary data. When the compiler determines that the output data is of type 1, it allocates a block from the currently "idle" Type 1 cache (convolution cache) pool.

[0078] "Second type data" specifically refers to data that needs to be retained, meaning the output of the operator will be referenced by subsequent branch structure operators. "Second type cache" specifically refers to the branch cache in this embodiment of the invention. This type of cache block may be small in size, but it is used to store data that needs to be used across layers. When the compiler determines that the output data is of the second type, it will allocate a block from the second type cache (branch cache) pool that is currently in the "free" state.

[0079] By classifying caches into first and second categories and allocating them with memory blocks of different sizes (first-category caches are typically larger to accommodate large feature maps in mainstream computing, while second-category caches can be smaller), the system avoids the memory waste caused by uniformly allocating the maximum size to all caches. This allows for the deployment of more complex neural network models with limited memory resources, greatly improving memory utilization efficiency.

[0080] Sub-step S22: Based on the cache index corresponding to the cache block, determine the cache index of the output cache of the operator.

[0081] In this embodiment of the invention, after the compiler successfully allocates a specific, free cache block for the current operator through step S21, it needs to formally inform the operator of this allocation result. This step involves the compiler writing the unique identifier (cache index) of the allocated cache block into the parameter list of the current operator. This process is not a complex calculation; its purpose is to bind abstract cache resources to specific computational tasks, thereby completing the final resource preparation before the operator executes.

[0082] By accurately encoding the compiler's static allocation decisions into the final operator stream file through cached indexes, the device can fully reproduce the allocation intent at compile time. At runtime, the device does not need any dynamic decisions; it only needs to "look up" the address by index, which completely eliminates the uncertainty caused by runtime memory allocation failures or address calculation errors.

[0083] Step 205: Determine the corresponding physical address for storing the operation result based on the cache index of each operator and the mapping relationship.

[0084] In this embodiment of the invention, the corresponding physical address is determined based on the cache index of each operator and the mapping relationship to store the operation result. This step occurs in real time during the inference execution phase on the device side. When an operator needs to be executed, the inference engine extracts the output cache index pre-allocated by the compiler from its parameter list. Subsequently, the engine uses this index as the key to query the "index-address" mapping table that was established during the initialization phase.

[0085] Step 206: Generate a cache status table to record the status of each cache. After assigning an output cache index to the operator, set the status of the cache to busy. In this embodiment of the invention, generating and updating the cache state table is a dynamic maintenance process that continuously occurs during the model compilation phase, simulating the lifecycle of cache resources. The compiler internally creates and maintains a cache state table that records the real-time status (idle or busy) of all cache blocks defined by the cache configuration information. Whenever an output cache block is successfully allocated for an operator, the compiler immediately marks the status of that allocated cache block as busy in the state table. This synchronous update operation signifies that the cache resource has been occupied and cannot be allocated to any other operator before it is released, thus logically ensuring that cached data will not be illegally overwritten by subsequent operator write operations.

[0086] Step 207: When it is detected that the allocated cache meets the release condition, the state of the cache is set to idle; the release condition includes that the cache is not the direct input cache of the next operator and that the data stored in the cache is not referenced by other branch structure operators.

[0087] In this embodiment of the invention, releasing and reusing the cache is a crucial step in the memory management simulation loop during the compilation phase. After allocating an output cache for the current operator and updating the state table each time, the compiler immediately and proactively scans the entire cache state table, checking all cache blocks whose current state is "busy". For each such cache block, the compiler determines whether it simultaneously satisfies the following two release conditions: It is not a direct input cache for the next operator: that is, the data in this cache will not be used in the immediately following computation step. The data it stores is not referenced by any subsequent branching operators: that is, the data in this cache will not be needed by any Shortcut or Route layer in the future. When both of these conditions are met, the compiler immediately resets the state of the cache block to free in the state table. This "release" action is completed during the compilation phase, signifying that the cache block can be reassigned to subsequent operators immediately, thus achieving circular reuse of memory resources.

[0088] The maximum cache required for a typical convolutional layer differs from that required for residual connections or channel concatenation layers. To save memory, cache types are categorized into convolutional caches and branch caches. The two caches with indices 0 and 1 are convolutional caches, while the others are branch caches. Cache states are either free or busy. The cache with index 0 is initialized to busy and serves as the input buffer for the first operator in the neural network model. All other caches are initialized to free. During model compilation by the AI ​​compiler, it determines whether the result of the current operator operation will be used by subsequent shortcut / route layers. Based on the neural network model structure, the entire neural network model structure is parsed during compilation to understand the input source of each operator. This information allows the compiler to determine whether the result of the current operator operation will be used by subsequent shortcut / route layers. If it is not used, a free convolutional cache is allocated to store the result; otherwise, a free branch cache is allocated to store the result, and the allocated cache state is set to busy. Update the status of each cache. If the following two conditions are met: (1) the cache is not used as the input buffer of the next operator; (2) the cache data is not used by the shortcut / route layer behind the network model, then set the cache status to free. Record the obtained cache index in the operator parameter list.

[0089] The memory allocation strategy allocates caches based on their state, but after retrieving a cache, only the cache index is recorded in the operator parameter list, not the cache state, because no further memory management is needed during forward inference of the model on the chip side. On the chip side, based on the AI ​​model structure, the number, size, and index of caches are set, and the system statically allocates memory. Each cache corresponds to a block of address space. For example, after statically allocating memory in the following code, the address `dev_cfg.layer_buffer[index]` can be obtained based on the index.

[0090] Reference Figure 4 The diagram shows a flowchart of a memory management optimization method provided by an embodiment of the present invention. Figure 4This demonstration describes how the AI ​​compiler parses and abstracts a neural network model into a sequence of operators, performs static memory allocation analysis, and ultimately generates an operator stream file containing cache indexes. First, a contiguous memory pool is allocated and an index-address mapping is established. Then, operators are executed sequentially. During execution, the device retrieves the cache address based on the cache index recorded in the operator parameter list; this step is crucial for achieving efficient, fragment-free memory access. Afterward, execution is routed to NPU acceleration or CPU software execution based on the operator type, and the results are stored in a designated cache. This process is repeated until all operators have been executed, and finally, the results are exported.

[0091] This invention discloses a memory management optimization method, apparatus, electronic device, and storage medium. The method includes: allocating a contiguous memory pool according to the cache required for the forward inference process of a neural network model; the memory pool includes multiple cache blocks, each with a corresponding cache index; establishing a mapping relationship between the cache index of the cache block and the physical address of the cache block; obtaining the operator sequence obtained by compiling the forward inference process of the neural network model and the operation results obtained by each operator operation in the operator sequence; allocating a cache index for each operator in the operator sequence; and determining the corresponding physical address to store the operation result according to the cache index and mapping relationship of each operator. By allocating a contiguous memory pool during the initialization phase and reusing pre-divided fixed cache blocks only through index reuse throughout the inference process, memory fragmentation is avoided, ensuring efficient and continuous memory utilization, and effectively preventing inference task failure due to the inability to allocate contiguous memory.

[0092] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0093] Reference Figure 5 The diagram shows a structural block diagram of a memory management optimization device provided in an embodiment of the present invention. The device includes the following modules: The memory pool allocation module 301 is used to allocate a contiguous memory pool according to the cache required for the forward inference process of the neural network model; the memory pool includes multiple cache blocks, and each cache block has a corresponding cache index; The mapping relationship establishment module 302 is used to establish the mapping relationship between the cache index of the cache block and the physical address of the cache block; The operator sequence acquisition module 303 is used to acquire the operator sequence obtained by compiling the forward inference process of the neural network model and the operation results obtained by each operator operation in the operator sequence; The cache index allocation module 304 is used to allocate a cache index for each operator in the operator sequence; The operation result storage module 305 is used to determine the corresponding physical address based on the cache index of each operator and the mapping relationship to store the operation result.

[0094] In some embodiments, the memory pool allocation module 301 includes: The configuration information acquisition submodule is used to acquire the cache configuration information corresponding to the neural network model; the cache configuration information includes the required number of caches, cache type, and cache size. The contiguous memory pool allocation submodule is used to allocate a contiguous memory pool according to the cache configuration information.

[0095] In some embodiments, the contiguous memory pool allocation submodule includes: The total memory determination unit is used to determine the total memory size based on the cache size and to obtain a continuous memory pool corresponding to the total memory size. The cache block partitioning unit is used to divide the contiguous memory pool into multiple cache blocks according to the cache quantity and cache type, assign a cache index to each cache block, and determine the physical address of each cache block in the contiguous memory pool.

[0096] In some embodiments, the cache index allocation module 304 includes: The cache block selection submodule is used to select a free cache block from the plurality of cache blocks as the output cache of the operator; The cache index determination submodule is used to determine the cache index of the output cache of the operator based on the cache index corresponding to the cache block.

[0097] In some embodiments, the plurality of cache blocks are divided into cache blocks for storing different types of data; the cache block selection submodule includes: A data type determination unit is used to determine the type of the output data of the operator; The output buffer determination unit is used to select a free buffer block from the buffer blocks corresponding to the type of the output data of the operator as the output buffer of the operator.

[0098] In some embodiments, the cache block type includes a first type of cache and a second type of cache; the first type of cache is used to store data that is not referenced by subsequent branch structure operators; the second type of cache is used to store data that will be referenced by subsequent branch structure operators. The output buffer determination unit includes: The output data determination subunit is used to select a free cache block from the first type of cache as the output cache of the operator if the output data of the operator is of the first type of data; and to select a free cache block from the second type of cache as the output cache of the operator if the output data of the operator is of the second type of data.

[0099] In some embodiments, the apparatus further includes: The cache status recording module is used to generate a cache status table to record the status of each cache. After assigning an output cache index to the operator, the status of the cache is set to busy. The cache state modification module is used to set the state of the cache to idle when it is detected that the allocated cache meets the release conditions; the release conditions include that the cache is not the direct input cache of the next operator and that the data stored in the cache is not referenced by other branch structure operators.

[0100] As the apparatus embodiment is basically similar to the method embodiment, it is described in a relatively simple manner. For relevant details, please refer to the description of the method embodiment.

[0101] This invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various processes of the memory management optimization method embodiments described above and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0102] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the memory management optimization method embodiments described above and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0103] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0104] Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present invention is not limited to performing functions in the order shown or discussed. It may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0106] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A memory management optimization method, characterized in that, The method includes: A contiguous memory pool is allocated according to the cache required for the forward inference process of the neural network model; the memory pool includes multiple cache blocks, each with a corresponding cache index; Establish a mapping relationship between the cache index of the cache block and the physical address of the cache block; Obtain the operator sequence obtained by compiling the forward inference process of the neural network model and the operation results obtained by each operator operation in the operator sequence; Assign a cache index to each operator in the operator sequence; The corresponding physical address is determined based on the cache index of each operator and the mapping relationship to store the operation result.

2. The memory management optimization method according to claim 1, characterized in that, The method of allocating a contiguous memory pool for cache allocation based on the forward inference process of the neural network model includes: Obtain the cache configuration information corresponding to the neural network model; the cache configuration information includes the required number of caches, cache type, and cache size; Based on the cache configuration information, allocate a contiguous memory pool.

3. The memory management optimization method according to claim 2, characterized in that, The step of allocating a contiguous memory pool according to the cache configuration information includes: The total memory size is determined based on the cache size, and a contiguous memory pool corresponding to the total memory size is obtained; Based on the cache quantity and cache type, the contiguous memory pool is divided into multiple cache blocks, a cache index is assigned to each cache block, and the physical address of each cache block in the contiguous memory pool is determined.

4. The memory management optimization method according to claim 1, characterized in that, Assigning a cache index to each operator in the operator sequence includes: Select a free cache block from the plurality of cache blocks as the output cache of the operator; Based on the cache index corresponding to the cache block, determine the cache index of the output cache of the operator.

5. The memory management optimization method according to claim 4, characterized in that, The multiple cache blocks are divided into cache blocks for storing different types of data; The step of selecting a free cache block from the plurality of cache blocks as the output cache of the operator includes: Determine the type of output data of the operator; Select an empty buffer block from the buffer blocks corresponding to the type of the operator's output data as the operator's output buffer.

6. The memory management optimization method according to claim 5, characterized in that, The cache blocks include a first type of cache and a second type of cache; the first type of cache is used to store data that is not referenced by subsequent branch structure operators; the second type of cache is used to store data that will be referenced by subsequent branch structure operators. The step of selecting a free buffer block from the buffer blocks corresponding to the type of the operator's output data as the operator's output buffer includes: If the output data of the operator is of the first type, then a free cache block is selected from the first type of cache as the output cache of the operator; If the output data of the operator is of the second type, then a free cache block is selected from the second type of cache as the output cache of the operator.

7. The memory management optimization method according to claim 4, characterized in that, The method further includes: A cache status table is generated to record the status of each cache. After assigning an output cache index to the operator, the status of the cache is set to busy. When it is detected that the allocated cache meets the release condition, the state of the cache is set to idle; the release condition includes that the cache is not the direct input cache of the next operator and that the data stored in the cache is not referenced by other branch structure operators.

8. A memory management optimization device, characterized in that, The device includes: The memory pool allocation module is used to allocate a contiguous memory pool according to the cache required for the forward inference process of the neural network model; the memory pool includes multiple cache blocks, each cache block has a corresponding cache index; The mapping relationship establishment module is used to establish the mapping relationship between the cache index of the cache block and the physical address of the cache block; The operator sequence acquisition module is used to acquire the operator sequence obtained by compiling the forward inference process of the neural network model and the operation results obtained by each operator operation in the operator sequence; A cache index allocation module is used to allocate a cache index for each operator in the operator sequence; The operation result storage module is used to determine the corresponding physical address to store the operation result based on the cache index of each operator and the mapping relationship.

9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the memory management optimization method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the memory management optimization method as described in any one of claims 1-7.

Images