Memory allocation optimization method and system based on neural network reasoning
By constructing a list of memory allocation sequences in a neural network and using the principle of minimal movement for address rearrangement, the problem of many memory fragments and low reuse efficiency in the existing memory allocation algorithm is solved, and more efficient memory utilization is achieved.
Patent Information
- Application Number
- CN202411751216.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-03-21
AI Technical Summary
Existing memory allocation algorithms are prone to memory fragmentation, resulting in low memory reuse efficiency.
By traversing all nodes of the neural network, recording inputs and outputs, and building a memory allocation sequence list based on the connection multiplexing relationship between nodes, address rearrangement is performed using the principle of minimum movement to optimize memory allocation.
It effectively reduces the generation of memory fragmentation, improves memory reuse efficiency, and reduces memory consumption.
Smart Images
Figure CN119668861A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of AI chip reasoning technology, and in particular to a memory allocation optimization method and system based on neural network reasoning. Background Art
[0002] Memory allocation is an important topic in the computer field, which requires efficient use of memory resources when the program is running. In the field of AI chip reasoning, due to the pursuit of extreme performance, power consumption and area, efficient memory allocation is crucial. This process is often completed through static memory analysis methods during the compilation stage of the model.
[0003] Common static memory allocation algorithms mainly include the following categories:
[0004] 1) The first fit algorithm starts scanning from the starting address of the available memory each time, finds the first free block that meets the current allocation size, and then allocates the required memory. When there are a large number of memory allocation requests, memory fragmentation is very likely to occur;
[0005] 2) The best fit algorithm scans the entire memory free space to find the free block that meets the current allocation requirements and has the closest space size. When there are a large number of memory allocation requests, it is still easy to generate memory fragmentation;
[0006] 3) The worst fit algorithm refers to selecting the largest free block to meet the memory allocation requirements each time. The core idea is to ensure that the remaining free space is as large as possible after each memory allocation. This algorithm can reduce the generation of fragments, but it will lead to an increase in the total static memory consumption and low memory reuse efficiency.
[0007] 4) Next adaptation algorithm: This algorithm is a variation of the first adaptation algorithm. It scans backward from the last allocated address to find the first free block that meets the current memory size requirement. It is essentially the same as method 1.
[0008] 5) Buddy algorithm, which divides the memory block into blocks of power of 2. When a memory request occurs, the block with the closest size is selected for allocation. This algorithm has high efficiency in memory allocation and free memory merging. However, since the allocated memory block is larger than the actual memory block, internal fragmentation is serious:
[0009] 6) Segmented and paging algorithm refers to managing memory by segmentation and paging, that is, managing memory in a refined manner through two-layer linked lists or other structures. It has high allocation efficiency and can effectively reduce memory fragmentation. However, the algorithm has high algorithm complexity and high system overhead when recycling memory, and is generally not used for memory allocation of neural networks.
[0010] At present, a common approach to memory allocation algorithms for neural networks is to analyze reusable relationships based on dependencies. If the address spaces of A and B are reusable, A and B are allocated the same memory block. If the sizes of A and B differ greatly, this method will also generate a large amount of memory fragmentation and a low memory reuse rate. The most intuitive manifestation is the high total memory consumption. It can be seen that the current mainstream memory allocation algorithm is still relatively easy to generate memory fragmentation, resulting in low memory reuse efficiency. Summary of the invention
[0011] The purpose of the present invention is to provide a memory allocation optimization method and system based on neural network reasoning to solve the problem that the existing memory allocation algorithm is prone to memory fragmentation, resulting in low memory reuse efficiency.
[0012] To achieve the above object, the present invention provides a memory allocation optimization method based on neural network reasoning, comprising:
[0013] Traverse all nodes of the neural network, record the input and output of each node, and determine the memory space to be allocated;
[0014] Construct a memory allocation sequence list based on the connection reuse relationship between the input and output of each node;
[0015] Allocate memory space to each node according to the memory allocation sequence list, and rearrange the addresses of the allocated memory space based on the minimum movement principle to optimize memory allocation, specifically including:
[0016] When allocating memory for the node, scan the free space list, and directly allocate when a free space that meets the allocation requirement and is closest in size is found;
[0017] When no free space that meets the allocation requirements is found, memory is allocated starting from the free space closest in size, the addresses of nodes with overlapping life cycles are rearranged, and the free space behind the node is compressed.
[0018] Optionally, when traversing all nodes of the neural network, a penetration analysis is performed on the nodes that do not produce data changes, and the input and output of the node after the penetration occurs are recorded.
[0019] Optionally, when constructing the memory allocation sequence list according to the connection reuse relationship between the input and output of each node, memory space is only allocated to the output of each node, that is, the output of the previous node serves as the input of the next node.
[0020] Optionally, the method for constructing the memory allocation sequence list includes:
[0021] Initialize the memory allocation sequence list and the list of nodes to be released;
[0022] Traversing the neural network, creating an allocation sequence for the current node and adding it to the memory allocation sequence list, and adding the current node to the list of nodes to be released;
[0023] The list of nodes to be released is traversed. If the outputs of the current node to be released have been allocated, a release sequence is created for the current node to be released and added to the memory allocation sequence list, and the list of nodes to be released is updated at the same time.
[0024] Optionally, for the same network segment, the size of the static space occupied by each node remains unchanged before and after address re-arrangement.
[0025] Based on the same technical concept, the present invention also provides a memory allocation optimization system based on neural network reasoning, comprising:
[0026] A traversal module is configured to traverse all nodes of the neural network, record the input and output of each node, and determine the memory space to be allocated;
[0027] A construction module, configured to construct a memory allocation sequence list according to a connection reuse relationship between inputs and outputs of each node;
[0028] An allocation module is configured to allocate memory space to each node according to the memory allocation sequence list, and to rearrange the addresses of the allocated memory space based on the minimum movement principle to optimize memory allocation. The allocation module is specifically configured as follows:
[0029] When allocating memory for the node, scan the free space list, and directly allocate when a free space that meets the allocation requirement and is closest in size is found;
[0030] When no free space that meets the allocation requirements is found, memory is allocated starting from the free space closest in size, the addresses of nodes with overlapping life cycles are rearranged, and the free space behind the node is compressed.
[0031] Optionally, the traversal module is further configured to:
[0032] When traversing all nodes of the neural network, a penetration analysis is performed on the nodes that do not generate data changes, and the input and output of the node after the penetration occurs are recorded.
[0033] Optionally, when the construction module constructs the memory allocation sequence list according to the connection reuse relationship between the input and output of each node, memory space is only allocated to the output of each node, that is, the output of the previous node serves as the input of the next node.
[0034] Optionally, the method in which the construction module constructs the memory allocation sequence list includes:
[0035] Initialize the memory allocation sequence list and the list of nodes to be released;
[0036] Traversing the neural network, creating an allocation sequence for the current node and adding it to the memory allocation sequence list, and adding the current node to the list of nodes to be released;
[0037] The list of nodes to be released is traversed. If the outputs of the current node to be released have been allocated, a release sequence is created for the current node to be released and added to the memory allocation sequence list, and the list of nodes to be released is updated at the same time.
[0038] Based on the same technical concept, the present invention also provides a readable storage medium on which a computer program is stored. When the computer program is executed, it can implement the memory allocation optimization method based on neural network reasoning as described above.
[0039] In a memory allocation optimization method and system based on neural network reasoning provided by the present invention, at least one of the following beneficial effects is achieved:
[0040] 1) Based on the characteristics of neural networks, a memory reuse strategy is formulated according to the life cycle of each node. The allocated memory is re-arranged based on the minimum movement principle, and the released memory is efficiently managed. This effectively reduces the generation of memory fragmentation and improves memory reuse efficiency.
[0041] 2) By performing penetration analysis on nodes in the neural network that do not generate data changes and recording the input and output after the penetration occurs, the memory overhead of such nodes can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Those skilled in the art will appreciate that the accompanying drawings are provided for a better understanding of the present invention and do not constitute any limitation on the scope of the present invention.
[0043] Figure 1 A step diagram of a memory allocation optimization method based on neural network reasoning provided by an embodiment of the present invention;
[0044] Figure 2 A schematic diagram of a network fragment including multi-layer concat provided by an embodiment of the present invention;
[0045] Figure 3 A flowchart of creating a memory allocation sequence list provided by an embodiment of the present invention;
[0046] Figure 4 A node diagram of a network segment provided by an embodiment of the present invention;
[0047] Figure 5A flowchart of memory allocation provided by an embodiment of the present invention;
[0048] Figure 6 A memory usage diagram before allocating memory block X provided by an embodiment of the present invention;
[0049] Figure 7 A diagram of memory usage after allocating memory block X provided by an embodiment of the present invention;
[0050] Figure 8 A structural block diagram of a memory allocation optimization system based on neural network reasoning provided by one embodiment of the present invention.
[0051] in:
[0052] 100-traverse module; 200-build module; 300-allocate module. DETAILED DESCRIPTION
[0053] In order to make the purpose, advantages and features of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, please refer to the accompanying drawings. It should be noted that the structure, proportion, size, etc. illustrated in the drawings of this specification are only used to match the content disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention. Any modification of the structure, change in the proportional relationship or adjustment of the size, under the same or similar conditions as the effects that can be produced by the present invention and the purposes that can be achieved, should still fall within the scope of the technical content disclosed by the present invention.
[0054] As used in the present invention, the singular forms "one", "an", and "the" include plural objects, unless the content clearly indicates otherwise. As used in the present invention, the term "or" is generally used in a sense that includes "and / or", unless the content clearly indicates otherwise. As used in the present invention, the term "several" is generally used in a sense that includes "at least one", unless the content clearly indicates otherwise. As used in the present invention, the term "at least two" is generally used in a sense that includes "two or more", unless the content clearly indicates otherwise. In addition, the terms "first", "second", and "third" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first", "second", and "third" may explicitly or implicitly include one or at least two of the features.
[0055] In the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", and "fixed" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0056] Please refer to Figure 1 , an embodiment of the present invention provides a memory allocation optimization method based on neural network reasoning, comprising the following steps:
[0057] S1, traverse all nodes of the neural network, record the input and output of each node, and determine the memory space to be allocated;
[0058] S2. Construct a memory allocation sequence list according to the connection reuse relationship between the input and output of each node;
[0059] S3, allocating memory space to each node according to the memory allocation sequence list, and re-arranging the addresses of the allocated memory space based on the minimum movement principle to optimize memory allocation, specifically including:
[0060] When allocating memory for the node, scan the free space list, and directly allocate when a free space that meets the allocation requirement and is closest in size is found;
[0061] When no free space that meets the allocation requirements is found, memory is allocated starting from the free space closest in size, the addresses of nodes with overlapping life cycles are rearranged, and the free space behind the node is compressed.
[0062] This embodiment is based on the characteristics of neural networks and formulates a memory reuse strategy according to the life cycle of each node. It rearranges the addresses of allocated memory based on the minimum movement principle and performs efficient idle management on the released memory, effectively reducing the generation of memory fragmentation and improving memory reuse efficiency.
[0063] First, S1 is executed to traverse all nodes of the neural network, record the input and output of each node, and determine the memory space to be allocated. The "node" mentioned in the present invention can also be called an "operator".
[0064] Preferably, when traversing all nodes of the neural network, a penetration analysis is performed on the nodes that do not produce data changes, and the input and output of the node after the penetration occurs are recorded.
[0065] This embodiment combines the characteristics of the neural network to perform penetration analysis on the nodes in the neural network that do not generate data changes, and records the input and output after the penetration occurs. If no penetration occurs, the input and output are recorded normally. Among them, the nodes that do not generate instructions mainly include reshape (tensor deformation), squeeze (dimensional deletion), expand_dims (dimensional increase), batch_flatten (dimensional expansion), and concat (dimensional splicing) and other nodes that have no data calculation and only deformation. The reshape, squeeze, expand_dims and batch_flatten nodes can be skipped directly when allocating memory. The processing of the concat node will become much more complicated. It should be noted that different chips have different levels of support for concat. Some concats will still generate instructions. This type of concat cannot penetrate. For example Figure 2 As shown in the figure, if concat_0 and concat_1 can be penetrated, the output of A is recorded as BCD, and the input of E is recorded as BCD. In the subsequent memory allocation, it is only necessary to ensure that the life cycle of BCD is consistent and the address space is continuous, that is, the total memory of BCD is allocated when B is allocated, and the total memory of BCD is released when D is released, so as to complete the memory allocation optimization of the concat node, thereby reducing the memory overhead.
[0066] Then execute S2 to build a memory allocation sequence list based on the connection reuse relationship between the input and output of each node.
[0067] The present invention constructs a new memory allocation model, which is implemented in two steps with memory allocation sequence as the basic unit. First, a memory allocation sequence list is constructed according to the network fragment, and then memory is allocated according to the memory allocation sequence list. For example, "["Alloc", "A", alloc_size]" means allocating memory space of size alloc_size to node A; "["Free", "A", alloc_size]" means releasing the memory space of node A of size alloc_size. This scheme decouples the memory allocation sequence list from the memory allocation algorithm. When constructing the memory allocation sequence list, an algorithm based on input-output dependency, an algorithm based on lifecycle analysis, or other conventional algorithms can be used. In this embodiment, in order to facilitate the memory allocation optimization of the concat node, an algorithm based on lifecycle analysis is used to construct the memory allocation sequence list.
[0068] Preferably, when constructing the memory allocation sequence list according to the connection reuse relationship between the input and output of each node, only the memory space is allocated for the output of each node, that is, the output of the previous node is used as the input of the next node. This can easily represent the connection reuse relationship between nodes. The reshape, squeeze, expand_dims and batch_flatten nodes can be directly skipped when allocating memory. If penetration optimization cannot be used for the concat node, it is treated as a normal node. If penetration optimization can be used, such as Figure 2 As shown, nodes B, C, and D will be analyzed for life cycle and memory allocation as a whole. After the memory allocation is completed, address values will be assigned to B, C, and D in turn. The creation process of the memory allocation sequence list is as follows: Figure 3 As shown, the details are as follows:
[0069] a) Initialize the memory allocation sequence list and the list of nodes to be released;
[0070] b) traversing the neural network, creating an allocation sequence for the current node and adding it to the memory allocation sequence list, and adding the current node to the list of nodes to be released;
[0071] c) traverse the list of nodes to be released. If the outputs of the current node to be released have been allocated, create a release sequence for the current node to be released and add it to the memory allocation sequence list, and update the list of nodes to be released. After the traversal in c is completed, return to b until the entire neural network traversal is completed.
[0072] by Figure 4 Taking the network fragment in as an example, the creation process and results of its memory allocation sequence list are as follows:
[0073] 1. Allocate A, ["Alloc", "A", alloc_size]
[0074] 2. Allocate B, ["Alloc", "B", alloc_size]
[0075] 3. Release A, ["Free", "A", alloc_size]
[0076] 4. Allocate C, ["Alloc", "C", alloc_size]
[0077] 5. Allocate D, ["Alloc", "D", alloc_size]
[0078] 6. Release C, ["Free", "C", alloc_size]
[0079] 7. Allocate E, ["Alloc", "E", alloc_size]
[0080] 8. Release B, ["Free", "B", alloc_size]
[0081] 9. Allocate F, ["Alloc", "A", alloc_size]
[0082] 10. Release D, ["Free", "D", alloc_size]
[0083] 11. Release E, ["Free", "E", alloc_size]
[0084] 12. Allocate G, ["Alloc", "G", alloc_size].
[0085] After the memory allocation sequence list is constructed, S3 is executed to allocate memory space to each node according to the memory allocation sequence list, and to rearrange the addresses of the allocated memory space based on the minimum movement principle to optimize memory allocation, specifically including:
[0086] When allocating memory for the node, scan the free space list, and directly allocate when a free space that meets the allocation requirement and is closest in size is found;
[0087] When no free space that meets the allocation requirements is found, memory is allocated starting from the free space closest in size, the addresses of nodes with overlapping life cycles are rearranged, and the free space behind the node is compressed.
[0088] In view of the characteristics of many memory fragments and poor memory reusability in the existing neural network memory allocation algorithm, the present invention proposes a memory allocation optimization strategy based on the minimum movement principle when re-arranging addresses. The core idea is to scan the free space list when allocating memory for the node, and directly allocate when the free space that meets the allocation requirements and is closest in size is found. At this time, the total memory usage remains unchanged. This step is similar to the best fit algorithm in the prior art; when no free space that meets the allocation requirements is found, memory is still allocated from the free space closest in size. If the direct allocation will trample on part of the allocated memory space (i.e., there is overlap). In order to solve this problem, it is necessary to re-arrange the addresses of the memory space of the nodes with overlapping life cycles, and continuously compress the subsequent free space, that is, compress the free space by moving the allocated space with the minimum movement.
[0089] In this embodiment, please refer to Figure 5 , the memory allocation process is as follows:
[0090] a) Initialize the array used_spaces to record the memory segments currently in use and the array free_sapces to record the memory segments that have been released.
[0091] b) Traverse each sequence in the memory allocation sequence list. If it is a sequence to be released, such as "["Free", "A", alloc_size]", then directly release the memory and update the free_spaces array; if it is an allocation sequence, such as "["Alloc", "A", alloc_size]", then look for suitable free space in free_spaces.
[0092] c) If free space that meets the allocation requirements is found in the free_spaces free array, space is allocated. If not found, address reordering and the memory allocation optimization strategy of the minimum movement principle are adopted. If the address reordering succeeds, memory is allocated. If it fails, memory is allocated in the new address space, which will increase the total size of the static memory. It can be found that if the address reordering fails, it degenerates into the best fit algorithm in the prior art introduction. In actual processing, address reordering can be tried separately after multiple free blocks. Through a large number of network memory allocation tests, it is found that the probability of finally triggering address reordering and reordering failure is very low.
[0093] Repeat the above process until the allocation sequence list traversal is completed. In this process, a very important point is how to judge whether the address reordering is successful. This can be achieved by ensuring that the addresses of nodes with overlapping life cycles cannot be stepped on. Figure 4 Take the memory allocation sequence list of the corresponding network segment as an example. For each node, Alloc indicates the beginning of the life cycle, and Free indicates the end of the life cycle. The life cycles of the nodes in the network segment are as follows: A: (1, 3), B: (2, 8), C: (4, 6), D: (5, 10), E: (7, 11), F: (9, 12), G: (12, 12). Currently, the execution reaches number 9 to allocate an address to F. If address reordering is triggered at this time and the address of D needs to be moved, to determine whether the new address of D can be successfully moved, it is only necessary to determine that all nodes that overlap with the life cycle (5, 10) of D, including B, C, E, and F, cannot trample on the new address space of D.
[0094] The following is a specific example to illustrate the memory allocation optimization strategy based on the minimum movement principle when reordering addresses proposed by the present invention. Assuming that the current memory usage is as follows Figure 6As shown in the figure, white represents free blocks and color fill represents allocated blocks. When a memory space of size 500 needs to be allocated, the conventional memory allocation algorithm can only allocate an additional memory block of size 500 in the new free space. At this time, the total static memory space size becomes 2100 (1600+500). After using the memory allocation optimization strategy based on the minimum movement principle proposed by the present invention, it is only necessary to allocate the free blocks of C to X, re-arrange the allocated memory space after C and continuously squeeze the subsequent free space. The allocation result is as follows: Figure 7 As shown, the addresses of D and F are rearranged, the total static space size is still 1600, and the memory fragmentation is reduced from four blocks to two blocks, which is better than the memory allocation algorithm mentioned above. Because each memory move requires an analysis of the life cycle and guarantees that the rearranged memory is not trampled, taking into account both algorithm complexity and total memory consumption, i.e., memory reuse efficiency, the present invention adopts the method of rearranging subsequent space from the free blocks with the closest size, which can effectively reduce memory fragmentation and improve memory reuse efficiency under low algorithm complexity. Combined with Figure 6 and Figure 7 , the specific process is as follows:
[0095] 1. Find the free block C[300, 600) with the closest size;
[0096] 2. Allocate space [300, 800) to the new node X and delete the free node C;
[0097] 3. Shift the first used block D on the right to [800, 1000);
[0098] 4. At this time, the free space of E = [800, 900) is completely covered, and the Free node E is deleted;
[0099] 5. Continue to translate the used block F on the right to [1000, 1200);
[0100] 6. At this time, there is still free space left at G = [1100, 1300), and the space movement ends here, and G is compressed to [1200, 1300).
[0101] Through a large number of tests, it was found that compared with the traditional memory reuse algorithm based on dependency analysis, the memory allocation optimization method based on the minimum movement principle provided by this embodiment reduces the total static memory consumption by about 20% on average, and the memory reuse efficiency is significantly improved.
[0102] Based on the same invention concept, please refer to Figure 8 The embodiment of the present invention further proposes a memory allocation optimization system based on neural network reasoning, comprising:
[0103] The traversal module 100 is configured to traverse all nodes of the neural network, record the input and output of each node, and determine the memory space to be allocated;
[0104] A construction module 200 is configured to construct a memory allocation sequence list according to a connection multiplexing relationship between inputs and outputs of each node;
[0105] The allocation module 300 is configured to allocate memory space to each node according to the memory allocation sequence list, and to rearrange the addresses of the allocated memory space based on the minimum movement principle to optimize the memory allocation. The allocation module 300 is specifically configured as follows:
[0106] When allocating memory for the node, scan the free space list, and directly allocate when a free space that meets the allocation requirement and is closest in size is found;
[0107] When no free space that meets the allocation requirements is found, memory is allocated starting from the free space closest in size, the addresses of nodes with overlapping life cycles are rearranged, and the free space behind the node is compressed.
[0108] Preferably, the traversal module 100 is further configured to:
[0109] When traversing all nodes of the neural network, a penetration analysis is performed on the nodes that do not generate data changes, and the input and output of the node after the penetration occurs are recorded.
[0110] This embodiment combines the characteristics of the neural network to perform penetration analysis on the nodes in the neural network that do not generate data changes, and records the input and output after the penetration occurs. If no penetration occurs, the input and output are recorded normally. Among them, the nodes that do not generate instructions mainly include reshape (tensor deformation), squeeze (dimensional deletion), expand_dims (dimensional increase), batch_flatten (dimensional expansion), and concat (dimensional splicing) and other nodes that have no data calculation and only deformation. The reshape, squeeze, expand_dims and batch_flatten nodes can be skipped directly when allocating memory. The processing of the concat node will become much more complicated. It should be noted that different chips have different levels of support for concat. Some concats will still generate instructions. This type of concat cannot penetrate. For example Figure 2As shown in the figure, if concat_0 and concat_1 can be penetrated, the output of A is recorded as BCD, and the input of E is recorded as BCD. In the subsequent memory allocation, it is only necessary to ensure that the life cycle of BCD is consistent and the address space is continuous, that is, the total memory of BCD is allocated when B is allocated, and the total memory of BCD is released when D is released, so as to complete the memory allocation optimization of the concat node, thereby reducing the memory overhead.
[0111] Preferably, when the construction module 200 constructs the memory allocation sequence list according to the connection reuse relationship between the input and output of each node, memory space is only allocated to the output of each node, that is, the output of the previous node is used as the input of the next node. In this way, the connection reuse relationship between nodes can be easily represented, and the reshape, squeeze, expand_dims and batch_flatten nodes can be directly skipped when allocating memory. If penetration optimization cannot be used for the concat node, it is treated as a normal node. If penetration optimization can be used, such as Figure 2 As shown, nodes B, C, and D will be analyzed for life cycle and memory allocation as a whole. After the memory allocation is completed, address values will be assigned to B, C, and D respectively.
[0112] Preferably, the method by which the construction module 200 constructs the memory allocation sequence list includes:
[0113] Initialize the memory allocation sequence list and the list of nodes to be released;
[0114] Traversing the neural network, creating an allocation sequence for the current node and adding it to the memory allocation sequence list, and adding the current node to the list of nodes to be released;
[0115] The list of nodes to be released is traversed. If the outputs of the current node to be released have been allocated, a release sequence is created for the current node to be released and added to the memory allocation sequence list, and the list of nodes to be released is updated at the same time.
[0116] In view of the characteristics of many memory fragments and poor memory reusability in the existing neural network memory allocation algorithm, the present invention proposes a memory allocation optimization strategy based on the minimum movement principle when re-arranging addresses. The core idea is to scan the free space list when allocating memory for the node, and directly allocate when the free space that meets the allocation requirements and is closest in size is found. At this time, the total memory usage remains unchanged. This step is similar to the best fit algorithm in the prior art; when no free space that meets the allocation requirements is found, memory is still allocated from the free space closest in size. If the direct allocation will trample on part of the allocated memory space (i.e., there is overlap). In order to solve this problem, it is necessary to re-arrange the addresses of the memory space of the nodes with overlapping life cycles, and continuously compress the subsequent free space, that is, compress the free space by moving the allocated space with the minimum movement.
[0117] In this embodiment, please refer to Figure 5 , the memory allocation process is as follows:
[0118] a) Initialize the array used_spaces to record the memory segments currently in use and the array free_sapces to record the memory segments that have been released.
[0119] b) Traverse each sequence in the memory allocation sequence list. If it is a sequence to be released, such as "["Free", "A", alloc_size]", then directly release the memory and update the free_spaces array; if it is an allocation sequence, such as "["Alloc", "A", alloc_size]", then look for suitable free space in free_spaces.
[0120] c) If free space that meets the allocation requirements is found in the free_spaces free array, space is allocated. If not found, address reordering and the memory allocation optimization strategy of the minimum movement principle are adopted. If the address reordering succeeds, memory is allocated. If it fails, memory is allocated in the new address space, which will increase the total size of the static memory. It can be found that if the address reordering fails, it degenerates into the best fit algorithm in the prior art introduction. In actual processing, address reordering can be tried separately after multiple free blocks. Through a large number of network memory allocation tests, it is found that the probability of finally triggering address reordering and reordering failure is very low.
[0121] Repeat the above process until the allocation sequence list traversal is completed. In this process, a very important point is how to judge whether the address reordering is successful. This can be achieved by ensuring that the addresses of nodes with overlapping life cycles cannot be stepped on. Figure 4Take the memory allocation sequence list of the corresponding network segment as an example. For each node, Alloc indicates the beginning of the life cycle, and Free indicates the end of the life cycle. The life cycles of the nodes in the network segment are as follows: A: (1, 3), B: (2, 8), C: (4, 6), D: (5, 10), E: (7, 11), F: (9, 12), G: (12, 12). Currently, the execution reaches number 9 to allocate an address to F. If address reordering is triggered at this time and the address of D needs to be moved, to determine whether the new address of D can be successfully moved, it is only necessary to determine that all nodes that overlap with the life cycle (5, 10) of D, including B, C, E, and F, cannot trample on the new address space of D.
[0122] Based on the same inventive concept, an embodiment of the present invention further proposes a readable storage medium on which a computer program is stored. When the computer program is executed, it can implement the memory allocation optimization method based on neural network reasoning as described above.
[0123] The readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device, such as but not limited to an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof. The more specific example (non-exhaustive list) of the readable storage medium includes: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer program described herein can be downloaded from the readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer program from the network and forwards the computer program for storage in the readable storage medium in each computing / processing device. The computer program for performing the operation of the present invention can be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, or a source code or object code written in any combination of one or more programming languages, including object-oriented programming languages-such as Smalltalk, C++, etc., and conventional procedural programming languages-such as "C" language or similar programming languages. The computer program can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by utilizing the state information of a computer program to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute computer-readable program instructions to implement various aspects of the present invention.
[0124] Here, various aspects of the present invention are described with reference to the flowchart and / or block diagram of the method, system and computer program product according to the embodiment of the present invention. It should be understood that each square frame of the flowchart and / or block diagram and the combination of the square frames in the flowchart and / or block diagram can be realized by a computer program. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so as to produce a machine, so that these programs, when executed by the processor of a computer or other programmable data processing device, produce a device for realizing the function / action specified in one or more square frames in the flowchart and / or block diagram. These computer programs can also be stored in a readable storage medium, and these computer programs make the computer, programmable data processing device and / or other equipment work in a specific way, so that the readable storage medium storing the computer program includes a manufactured product, which includes instructions for realizing various aspects of the function / action specified in one or more square frames in the flowchart and / or block diagram.
[0125] The computer program may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are executed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the computer program executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0126] In summary, the embodiment of the present invention provides a memory allocation optimization method and system based on neural network reasoning. Based on the characteristics of the neural network, a memory reuse strategy is formulated according to the life cycle of each node, and the allocated memory is re-arranged based on the minimum movement principle. At the same time, the released memory is efficiently managed for idleness, which effectively reduces the generation of memory fragmentation and improves the efficiency of memory reuse. In addition, by performing penetration analysis on nodes in the neural network that do not generate data changes and recording the input and output after penetration occurs, the memory overhead of such nodes can be reduced.
[0127] The above description is only a description of the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any changes and modifications made by a person skilled in the art in the field of the present invention based on the above disclosure are within the scope of protection of the present invention. Obviously, a person skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A memory allocation optimization method based on neural network reasoning, characterized in that: include: Traverse all nodes of the neural network, record the input and output of each node, and determine the memory space to be allocated; Construct a memory allocation sequence list based on the connection reuse relationship between the input and output of each node; Allocate memory space to each node according to the memory allocation sequence list, and rearrange the addresses of the allocated memory space based on the minimum movement principle to optimize memory allocation, specifically including: When allocating memory for the node, scan the free space list, and directly allocate when a free space that meets the allocation requirement and is closest in size is found; When no free space that meets the allocation requirements is found, memory is allocated starting from the free space closest in size, the addresses of nodes with overlapping life cycles are rearranged, and the free space behind the node is compressed.
2. The memory allocation optimization method based on neural network reasoning according to claim 1 is characterized in that: When traversing all nodes of the neural network, a penetration analysis is performed on the nodes that do not generate data changes, and the input and output of the node after the penetration occurs are recorded.
3. The memory allocation optimization method based on neural network reasoning according to claim 1 is characterized in that: When the memory allocation sequence list is constructed according to the connection reuse relationship between the input and output of each node, memory space is only allocated to the output of each node, that is, the output of the previous node serves as the input of the next node.
4. The memory allocation optimization method based on neural network reasoning according to claim 3 is characterized in that: The method for constructing the memory allocation sequence list includes: Initialize the memory allocation sequence list and the list of nodes to be released; Traversing the neural network, creating an allocation sequence for the current node and adding it to the memory allocation sequence list, and adding the current node to the list of nodes to be released; The list of nodes to be released is traversed. If the outputs of the current node to be released have been allocated, a release sequence is created for the current node to be released and added to the memory allocation sequence list, and the list of nodes to be released is updated at the same time.
5. The memory allocation optimization method based on neural network reasoning according to claim 1 or 4, characterized in that: For the same network segment, the size of the static space occupied by each node remains unchanged before and after address re-arrangement.
6. A memory allocation optimization system based on neural network reasoning, characterized in that: include: A traversal module is configured to traverse all nodes of the neural network, record the input and output of each node, and determine the memory space to be allocated; A construction module, configured to construct a memory allocation sequence list according to a connection reuse relationship between inputs and outputs of each node; An allocation module is configured to allocate memory space to each node according to the memory allocation sequence list, and to rearrange the addresses of the allocated memory space based on the minimum movement principle to optimize memory allocation. The allocation module is specifically configured as follows: When allocating memory for the node, scan the free space list, and directly allocate when a free space that meets the allocation requirement and is closest in size is found; When no free space that meets the allocation requirements is found, memory is allocated starting from the free space closest in size, the addresses of nodes with overlapping life cycles are rearranged, and the free space behind the node is compressed.
7. The memory allocation optimization system based on neural network reasoning according to claim 6 is characterized in that: The traversal module is also configured to: When traversing all nodes of the neural network, a penetration analysis is performed on the nodes that do not generate data changes, and the input and output of the node after the penetration occurs are recorded.
8. The memory allocation optimization system based on neural network reasoning according to claim 6 is characterized in that: When the construction module constructs the memory allocation sequence list according to the connection reuse relationship between the input and output of each node, the memory space is only allocated to the output of each node, that is, the output of the previous node is used as the input of the next node.
9. The memory allocation optimization system based on neural network reasoning according to claim 6 is characterized in that: The method for constructing the memory allocation sequence list by the construction module includes: Initialize the memory allocation sequence list and the list of nodes to be released; Traversing the neural network, creating an allocation sequence for the current node and adding it to the memory allocation sequence list, and adding the current node to the list of nodes to be released; The list of nodes to be released is traversed. If the outputs of the current node to be released have been allocated, a release sequence is created for the current node to be released and added to the memory allocation sequence list, and the list of nodes to be released is updated at the same time.
10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, it can implement the memory allocation optimization method based on neural network reasoning according to any one of claims 1-5.