Memory offset calculation method based on depth sorting and tensor activeness priority

Abstract

The invention provides a memory offset calculation method based on depth sorting and tensor activeness priority, and the method is used for static memory planning, and comprises the following steps: S1, determining a node scheduling sequence based on a hub tracing depth-first strategy; s2, sorting the distribution priorities of the intermediate tensors of the calculation graph through the tensor activeness; and S3, searching a memory gap, and searching a proper memory gap to match a corresponding intermediate tensor. According to the method, the node sequence is scheduled based on the depth-first strategy of hub tracing, the tensor active length is introduced into the memory offset calculation strategy, memory fragments can be solved to a certain extent, the size of a memory peak value can be reduced to a certain extent, and the problem of memory fragments can be solved to a certain extent, and the size of the memory peak value can be reduced to a certain extent.

Description

Technical Field

[0001] This invention belongs to the field of deep learning compiler optimization technology, and specifically relates to a memory offset calculation method based on depth sorting and tensor activity priority. Background Art

[0002] With the widespread application of deep learning technology in various fields, including computer vision and natural language processing, and the latest advancements in computing hardware, the inference tasks of deep neural networks are no longer limited to the server side. More and more deep neural networks are allowing inference tasks to be transferred to mobile and embedded devices.

[0003] At the same time, the application of deep learning tasks on mobile and embedded devices has led to the following challenges:

[0004] 1) Computational resource limitations: Mobile and embedded devices typically have limited computing resources, such as CPU, GPU, and memory. This can lead to insufficient computing power during the inference process of deep learning models.

[0005] 2) Storage space limitations: Mobile and embedded devices also have relatively limited storage space, which limits the size and number of parameters of deep learning models.

[0006] 3) Energy Consumption Constraints: Mobile and embedded devices typically rely on batteries for power, thus having high energy consumption requirements. Deep learning models, on the other hand, often require significant computational and storage resources for inference, consuming substantial amounts of energy. Therefore, deep learning models need to minimize energy consumption while maintaining performance.

[0007] To address these challenges, deep learning compilers can optimize the computation graph, which helps reduce memory usage and accelerate computation. An optimized computation graph can better utilize hardware resources, such as GPUs and CPUs, thereby reducing memory accesses and improving computational efficiency. Secondly, deep learning compilers can leverage memory optimization techniques to reduce memory usage. For example, compilers can manage memory through techniques such as memory sharing and memory reclamation, avoiding problems such as memory leaks and memory overflows.

[0008] The goal of static memory planning optimization is to reuse memory buffers as much as possible. There are generally two methods: in-place memory sharing and standard memory sharing. In-place memory sharing uses the same memory for both input and output operations and allocates only one memory location before computation. Standard memory sharing reuses non-overlapping memory from previous operations. Static memory planning is done offline, which allows for the application of more complex planning algorithms.

[0009] However, the irregular computational patterns of deep learning networks, composed of interconnected nodes, complicate their execution. This irregularity leads to a new problem: during inference, the scheduling and order of nodes significantly impact memory usage. To enable neural networks to be applied to edge devices, a reasonable node scheduling order is needed.

[0010] In existing technologies, depth-first sorting is the most commonly used strategy for node scheduling in deep learning compilers, and its advantages are:

[0011] 1. Reduce redundant calculations: Depth-first sorting can effectively eliminate redundant calculations in the computation graph, thereby improving computational efficiency.

[0012] 2. Improve computation speed: Since depth-first sorting can eliminate redundant calculations, it can speed up the execution efficiency of the model.

[0013] 3. Easy to parallelize: Depth-first sorting can be easily combined with parallel computing to improve computational efficiency.

[0014] However, it also has some potential drawbacks:

[0015] 1. Prone to getting trapped in local conditions: Depth-first sorting may cause some nodes to be scheduled too early, while other nodes remain in a state of waiting for computation for a long time, which may exceed the limit of computing resources.

[0016] 2. May produce misleading results: Since the depth-first search process is unidirectional, once a branch cannot continue, the efficiency of the entire sorting process will decrease, resulting in a larger memory peak. Summary of the Invention

[0017] To address the aforementioned issues, the purpose of this application is to provide efficient memory management for deep neural networks, thereby optimizing memory usage and computational efficiency.

[0018] This method, regarding the overall concept:

[0019] The depth-first strategy for scheduling nodes based on hub tracing, and the introduction of tensor activity into the memory offset calculation strategy, can solve the problem of memory fragmentation to a certain extent and reduce the size of memory peaks.

[0020] Specifically, this invention provides a memory offset calculation method based on depth sorting and tensor liveness priority. For static memory planning, the method includes the following steps:

[0021] S1. The node scheduling order is determined based on the depth-first strategy of hub tracing, that is, the node scheduling order is obtained by setting hub nodes and adding tracing function. The tracing function is the process of obtaining the set of ordinary nodes that the hub nodes depend on in reverse and prioritizing the sorting of their dependencies. This process includes two parts: forward sorting and reverse tracing, which complement each other.

[0022] S2, sort the allocation priority of intermediate tensors in the computation graph by tensor active length; the active length of tensor indicates that tensor must exist in memory for a certain period of time, and reflects its active state in memory to a certain extent; that is, the memory calculation offset strategy based on tensor activity can ensure that tensors in similar time periods are as close as possible in memory.

[0023] S3, find memory gaps, and match the corresponding intermediate tensor when a suitable memory gap is found;

[0024] Memory gaps can be viewed as memory buffers that have been used before a certain point in time and are currently in a freed state;

[0025] For each intermediate tensor, first check the allocated tensors whose lifetimes intersect with the current tensor to find the smallest memory gap between them, so that the current tensor fits the gap; if such a gap is found, the current tensor is allocated to this gap; otherwise, the current tensor is allocated after the bottommost tensor whose lifetime intersects with its current tensor; the corresponding offset is assigned to the current tensor, and the tensor is in the allocated state.

[0026] In step S1, a hub node is defined as a node in the neural network with an in-degree greater than 1, and constant nodes are not considered in the process of counting the in-degree of nodes;

[0027] First, in a neural network, "in-degree" refers to the number of connections a node receives; for a hub node, its in-degree is greater than 1, which means that multiple other nodes send it information or activation signals.

[0028] Secondly, constant nodes are not considered when calculating the in-degree of a node; constant nodes are those nodes that hold static data, that is, those that always output the same data stream; they are not affected by other nodes; therefore, these constant nodes are excluded when calculating the in-degree of a hub node.

[0029] Step S1 further includes:

[0030] S1.1 Construct the scheduling list:

[0031] First, construct the scheduled list and the unscheduled list separately;

[0032] In this process, the scheduled list refers to the list of nodes or data that have been arranged according to a certain rule or order.

[0033] The scheduling list contains nodes that have not yet been sorted according to specific rules. In the initial stage of a neural network, a scheduling list is usually generated first, which contains all the nodes that need to be processed. As the processing progresses, some nodes are added to the scheduled list, and the scheduling list is continuously updated until all nodes have been processed.

[0034] Importing the S1.2 starting node:

[0035] In this step, the first step is to determine the starting node of the neural network. The starting node is usually the input layer node of the neural network, which is responsible for receiving raw data or features. Importing the starting node means loading these input layer nodes into a sorted list and preparing them for subsequent inference.

[0036] S1.3 Determine the hub node:

[0037] Next, we need to determine the hub nodes in the neural network. According to the previously defined definition, hub nodes are those nodes with an in-degree greater than 1, and constant nodes are not considered when calculating the in-degree. Hub nodes can be determined by traversing the entire neural network and calculating the in-degree of each node. During the traversal, constant nodes can be skipped, focusing only on nodes with multiple input connections. S1.4 Forward Depth-First Search:

[0038] Once the pivot node is identified, a forward depth-first search can begin. Forward depth-first search is a strategy for traversing a neural network. It starts from the initial node and explores each branch as deeply as possible along the network's connections, until the pivot node or leaf node is reached and further exploration is impossible. During the forward depth-first search process, the activation state and calculation results of each node can be recorded to provide a basis for subsequent analysis and processing.

[0039] Reverse tracing of the S1.5 hub node:

[0040] During the forward depth-first search process, if a hub node is encountered, it needs to be traced back. Tracing back means finding the node with input connectivity to the hub node from the list of nodes to be scheduled. Through tracing back, we can understand how the hub node is affected by input data or other nodes, as well as their role and scope of influence in the network.

[0041] S1.6 completes the hub node:

[0042] After the reverse tracing is completed, the hub nodes that have been processed need to be marked as "completed" to indicate that the analysis and processing of the node have been completed; then, the forward depth-first search can continue to be performed until the entire neural network has been traversed or other termination conditions are met.

[0043] Steps S1.4 to S1.6 described in S1.7 are repeated until the list of nodes to be scheduled is empty, at which point the network ends. Throughout the entire processing flow, connectivity is maintained and affects the selection of nodes and the processing order.

[0044] Step S1 further includes:

[0045] S1.1 Construct the scheduling list:

[0046] Assume that the scheduled list is empty in the initial state, and the list to be scheduled is {A,B,C,D,E,F,G,H,I,J,K,L};

[0047] Importing the S1.2 starting node:

[0048] Suppose that input layer node A is imported into the scheduled list, at which point the scheduled list is {A} and the list to be scheduled is {B,C,D,E,F,G,H,I,J,K,L}.

[0049] S1.3 Determine the hub node:

[0050] Suppose that the hub nodes in the neural network are F, J, and K, since their in-degrees are 2, 4, and 2, respectively;

[0051] S1.4 Forward Depth-First Search:

[0052] Assume that the forward depth-first search starts from the starting node A, proceeds sequentially along node B, node C, until it encounters the hub node F, at which point the forward process stops; at this point, the scheduled list is {A,B,C}, and the list to be scheduled is {D,E,F,G,H,I,J,K,L}.

[0053] Reverse tracing of the S1.5 hub node:

[0054] Suppose that when a hub node F is encountered, it is traced back to its dependencies D and E. Then a sub-scheduling list is created for the hub node's dependencies. The sub-scheduling list is {D,E}. The sub-scheduling list is iterated through S1.2-S1.5. At this time, the scheduled list is {A,B,C,D,E} and the list to be scheduled is {F,G,H,I,J,K,L}.

[0055] S1.6 completes the hub node:

[0056] Suppose that the currently scheduled list is {A,B,C,D,E,F}, and the list to be scheduled is {G,H,I,J,K,L}.

[0057] Step S1.7 further includes:

[0058] In step S.1.4, during the forward depth-first search, when a hub node J is encountered, in step S.1.5, during the reverse tracing, the node G that it depends on is added to the sub-schedulable list that hub node J depends on.

[0059] The sub-list to be scheduled is {H,G,I}. The sub-list to be scheduled is then subjected to the forward depth sorting operation in step S1.4. At this point, the already scheduled list is {A,B,C,D,E,F,H,G,I}, and the list to be scheduled is {J,K,L}. Step S1.6 completes the hub node process. At this point, the already scheduled list is {A,B,C,D,E,F,H,G,I,J}, and the list to be scheduled is {K,L}.

[0060] In step S.1.4, a forward depth-first search is performed. When a hub node K is encountered, step S.1.5 traces back to the child scheduling list that the hub node K depends on, which is added to the list of children to be scheduled that the hub node K depends on.

[0061] The sub-list to be scheduled is {L}. Perform the forward depth sorting operation in step S1.4 on the sub-list to be scheduled. At this time, the scheduled list is {A,B,C,D,E,F,H,G,I,J,L}, and the list to be scheduled is {K}. Step S1.6 completes the hub node. At this time, the scheduled list is {A,B,C,D,E,F,H,G,I,J,L,K}, and the list to be scheduled is {}. The sorting ends.

[0062] In step S2

[0063] The rule for ranking tensor activity is to sort them in a non-increasing manner according to indicators such as tensor activity length, tensor size, and starting point; the earlier the indicator appears, the higher its priority.

[0064] The three metrics in tensor activity ranking include: tensor activity length, tensor size, and starting point. 1. Tensor activity length

[0065] Tensor active length refers to the time period from when a tensor is created in a program until it is no longer used; it reflects the lifespan of a tensor in memory.

[0066] A longer active length means that tensors need to occupy memory for a longer time, which is more likely to cause memory bottlenecks or fragmentation problems. When sorting, prioritizing the allocation and release of tensors with shorter active lengths helps to reduce memory usage and fragmentation, and improve memory utilization efficiency.

[0067] 2. Tensor Dimensions

[0068] Tensor size refers to the amount of storage space occupied by a tensor, which is usually determined by its data type, dimensions, and number of elements;

[0069] Larger tensor sizes mean more memory space is needed to store them, which makes them more prone to memory shortages or fragmentation. When sorting, considering tensor size can help optimize memory allocation and avoid mixing large and small tensors, thereby reducing memory fragmentation.

[0070] 3. Starting point

[0071] The starting point refers to the creation time or the beginning of the lifecycle of a tensor in a program;

[0072] In some cases, understanding the order in which tensors are created can help predict their subsequent memory usage patterns, thus enabling more precise memory management; this includes the fact that some algorithms or operations may depend on a specific tensor creation order, and considering the starting point can better meet these needs.

[0073] During tensor sorting, tensors are first sorted according to their active length to ensure that tensors with longer active lengths are processed first. Then, among tensors with the same active length, tensors with larger sizes are processed first. Finally, if the first two criteria are the same, tensors with earlier starting points are processed first.

[0074] The tensor active length sorting is assumed to be as shown in the following table:

[0075]

[0076] Tensor activity sorting: 2->3->0->1->5->4->6;

[0077] During the tensor activity sorting process, the tensors are first sorted according to their activity length to ensure that tensors with longer activity lengths are processed first. Assuming that the activity length of Tensor 2 is 3 and the activity length of Tensor 3 is 2, Tensor 2 will be before Tensor 3 after sorting.

[0078] Then, among tensors with the same active length, ensure that the tensor with the larger tensor size is processed first. For example, if the tensor size of Tensor 3 is 5 and the tensor size of Tensor 0 is 4, then after sorting, Tensor 3 will be before Tensor 0.

[0079] Finally, if the first two metrics are the same, ensure that tensors with earlier starting points are processed first.

[0080] In step S2, the tensor activity offset calculation strategy is as follows:

[0081] A large block of memory is pre-allocated, and the intermediate tensor is a data buffer partitioned by offsets within the memory block. This method is called memory offset calculation, and its goal is to minimize the size of the allocated memory block.

[0082] Offset calculation can be viewed as a special case of a two-dimensional strip filling problem. It can be abstracted and simplified into a filling problem in which a set of rectangular strips with fixed coordinates in a certain dimension enter a container and their size is minimized by adjusting another dimension. If the height of the container represents the temporal domain of memory allocation, then the width of the container represents the memory usage.

[0083] In step S2, the intermediate tensor of the computation graph has the following characteristics:

[0084] Lifecycle: The lifecycle of a tensor represents the entire process during which the tensor is active in memory;

[0085] The lifetime of an intermediate tensor t can be defined as {start point, end point}, where the start point and end point are the producer operator of the intermediate tensor t and the index of the last operator that took the intermediate tensor t as its input, respectively. These indices come from a topological sorting of the neural network, which is also the execution order of the operators. It is worth noting that no two tensors with intersecting intervals can share memory.

[0086] In step S3, for an intermediate tensor Ta to be allocated memory:

[0087] 1) Calculate the required memory size S based on its data type, data layout, dimensions, and other information. a ;

[0088] 2) Traverse the list of allocated tensors and obtain the set V of tensors whose lifetimes intersect with the intermediate tensor to be allocated. t And sort the tensor set V in ascending order of its starting address. t Sort;

[0089] 3) Through tensor set V t Retrieve memory gap list V f Compare the size of each memory gap and denote it as f0, f1, ..., f n Find the first tensor T whose size is greater than or equal to its size. a The required memory and the optimal memory gap, where the optimal fit principle is min(f) x -S a )>0;

[0090] If a suitable memory gap V is found x To satisfy tensor T a The memory requirements are allocated to tensor T. a That is, tensor T a The starting address is V x The starting address;

[0091] If a sufficiently large memory gap cannot be found, the memory expansion function is used to expand the tensor T. a Assigned to tensor set V t The last tensor T b Following that, namely tensor T a The starting address size is tensor T b The starting address and tensor T b Add up the required memory sizes;

[0092] 5) Transfer the intermediate tensor T to be allocated memory. a Add to the list of assigned tensors.

[0093] Therefore, the advantage of this application is:

[0094] The depth-first sorting method based on hub tracing proposed in this application, while inheriting the existing advantages of depth-first sorting, can solve the problems of being prone to getting stuck in local conditions and producing misleading results.

[0095] This application proposes a tensor activity ranking that comprehensively considers three indicators: tensor activity length, tensor size, and starting point. This ranking can more effectively manage and optimize memory, thereby improving program performance and stability. Attached Figure Description

[0096] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0097] Figure 1 This is a schematic diagram of a depth-first process based on hub tracing.

[0098] For ease of explanation, Figure 1 Add numbers to the smaller images in the following order: (a), (b), (c), (d), (e), (f), (g):

[0099] Figure 1 (a) represents the initial state.

[0100] Figure 1 (b) is a schematic diagram of the relevant operations when encountering hub node F. Import the starting node A to perform a depth-first search until the hub node F is encountered, mark the child scheduling list that hub node F depends on, and perform a depth-first search on the child scheduling list.

[0101] Figure 1 (c), Figure 1 (d), Figure 1 (e) is a schematic diagram of the relevant operations when encountering hub node J. The sub-schedulable list that hub node J depends on is marked, and a depth-first search is performed on the sub-schedulable list.

[0102] Figure 1 (f) is a schematic diagram of the operations related to encountering hub node K, marking the sub-schedulable list that hub node K depends on, and performing a depth-first search on the sub-schedulable list.

[0103] Figure 1 (g) To complete the search of the entire network.

[0104] Figure 2 This is a schematic diagram of the topology of the computation flow graph.

[0105] Figure 3 This is a schematic diagram of the lifecycle and active length of a tensor.

[0106] Figure 4 This is a diagram illustrating the search for memory gaps.

[0107] Figure 5 This is a flowchart illustrating the method. Detailed Implementation

[0108] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0109] This invention belongs to the field of deep learning compilers and is applied to memory management optimizations in the compiler backend.

[0110] Effectively reusing memory buffers can improve cache hit rate and inference speed.

[0111] Therefore, a strategy for deep sorting and tensor activity offset calculation based on hub tracing is proposed:

[0112] A large block of memory is pre-allocated, and the intermediate tensor is a data buffer partitioned by offsets within the memory block. This method is called memory offset calculation, and its goal is to minimize the size of the allocated memory block.

[0113] The offset calculation problem can be viewed as a special case of the two-dimensional strip filling problem. This problem can be abstracted and simplified into a filling problem where a set of rectangular strips with fixed coordinates in one dimension enter a container, and their size is minimized by adjusting another dimension. If the height of the container represents the temporal domain of memory allocation, then the width of the container represents the memory usage.

[0114] The active length of a tensor indicates the duration during which the tensor must exist in memory, reflecting its active state in memory to some extent. Therefore, a strategy for calculating memory offsets based on tensor active length was explored. This strategy ensures that tensors within similar time periods are kept as close as possible in memory, and it also achieves good results for chained structures.

[0115] like Figure 5 As shown, a memory offset calculation method based on depth sorting and tensor liveness priority is provided. For static memory planning, the method includes the following steps:

[0116] S1, determine the node scheduling order based on the depth-first strategy of hub tracing, that is, the depth sorting based on hub tracing;

[0117] This application analyzes the drawbacks of the depth-first sorting method and achieves a better node scheduling order by setting hub nodes and adding a tracing function, thereby optimizing the memory usage of the inference engine. The process involves comparing the resources required by the hub nodes and then tracing to acquire as many hub node resources as possible. The tracing function retrieves the set of ordinary nodes that the hub nodes depend on from the back of the hub nodes and prioritizes their dependencies. This process includes both forward sorting and reverse tracing, which complement each other.

[0118] In step S1, the hub node is a crucial concept. Its primary function is to connect and integrate various dynamic data streams. Hub nodes are those nodes with very high connectivity in a neural network. Hub nodes possess aggregation and connection attributes. They are not only data distribution centers but also the central link connecting various network nodes. The functions of hub nodes will be analyzed in detail below from several aspects:

[0119] 1) From a network topology perspective, hub nodes typically refer to nodes with very high connectivity within a network. These nodes are similar to a data center. Their role is to aggregate and distribute data from different locations and along different paths, enabling fast and efficient data transmission.

[0120] 2) Hub nodes possess high stability because they connect a large number of network nodes and handle a significant amount of data transmission. A failure in a hub node can disrupt the normal operation of the entire network, potentially leading to issues such as memory overflow or leaks.

[0121] 3) Hub nodes possess powerful data transmission and integration capabilities. They can not only store large amounts of data, but also analyze and extract data.

[0122] In summary, this method defines a pivot node as a node in the neural network with an in-degree greater than 1, and does not consider constant nodes when calculating the in-degree of nodes; the following is a description of pivot nodes:

[0123] First, in a neural network, "in-degree" refers to the number of connections a node receives. For a hub node, its in-degree is greater than 1, meaning that multiple other nodes send it information or activation signals. Such nodes play a crucial role in neural networks because they can integrate information from different sources and significantly influence the network's output. In some complex tasks, hub nodes can play a key role in helping the network identify important patterns or features.

[0124] Secondly, constant nodes are not considered when calculating the in-degree of a node; constant nodes are those that hold static data (always outputting the same data stream); they are not affected by other nodes. Therefore, it is reasonable to exclude these constant nodes when calculating the in-degree of a hub node.

[0125] Furthermore, the main implementation steps of step S1 are as follows:

[0126] S1.1 Construct the scheduling list:

[0127] First, construct the scheduled list and the unscheduled list respectively.

[0128] In this process, the scheduled list typically refers to a list of nodes or data that have been arranged according to a specific rule or order. Generating the scheduled list helps us efficiently traverse the network and prioritize nodes that have a greater impact on overall performance or results.

[0129] The scheduling list contains nodes that have not yet been sorted according to specific rules. In the initial stages of a neural network, we typically generate a scheduling list containing all nodes that need to be processed. As processing progresses, some nodes are added to the scheduled list, and the scheduling list is continuously updated until all nodes have been processed.

[0130] exist Figure 1 In the initial state of the example, the scheduled list is empty, and the list to be scheduled is {A,B,C,D,E,F,G,H,I,J,K,L}.

[0131] Importing the S1.2 starting node:

[0132] In this step, the first step is to determine the starting nodes of the neural network. Starting nodes are typically the input layer nodes of the neural network; they are responsible for receiving raw data or features. Importing starting nodes means loading these input layer nodes into a sorted list, preparing them for subsequent inference.

[0133] exist Figure 1 In the example, input layer node A is imported into the scheduled list. At this point, the scheduled list is {A}, and the list to be scheduled is {B,C,D,E,F,G,H,I,J,K,L}.

[0134] S1.3 Determine the hub node:

[0135] Next, we need to identify the hub nodes in the neural network. According to the previously defined definition, hub nodes are those nodes with an in-degree greater than 1, and constant nodes are not considered when calculating the in-degree. Hub nodes can be determined by traversing the entire neural network and calculating the in-degree of each node. During the traversal, constant nodes can be skipped, focusing only on nodes with multiple input connections.

[0136] exist Figure 1 In the example, the hub nodes in the neural network can be identified as F, J, and K, because their in-degrees are 2, 4, and 2, respectively.

[0137] S1.4 Forward Depth-First Search:

[0138] Once the hub node is identified, we can begin a forward depth-first search. Forward depth-first search is a strategy for traversing a neural network. Starting from the initial node, it explores each branch as deeply as possible along the network's connections until it reaches the hub node or a leaf node where further exploration is impossible. During the forward depth-first search process, we can record the activation state and computation results of each node, providing a basis for subsequent analysis and processing.

[0139] exist Figure 1 In the example, the forward depth-first search starts from node A, proceeds sequentially along node B, node C, until it encounters the hub node F, at which point the forward process stops. At this point, the scheduled list is {A,B,C}, and the list to be scheduled is {D,E,F,G,H,I,J,K,L}.

[0140] Reverse tracing of the S1.5 hub node:

[0141] During the forward depth-first search process, if a hub node is encountered, it needs to be traced back. Tracing back involves finding nodes with input connectivity to the hub node from the list of nodes to be scheduled. Through tracing back, we can understand how the hub node is affected by input data or other nodes, and their role and scope of influence within the network.

[0142] exist Figure 1 In the example, when a hub node F is encountered, tracing back to its dependencies on nodes D and E, a sub-scheduling list is created based on the hub node's dependencies. The sub-scheduling list is {D, E}. The process of iterating through the sub-scheduling list from S1.2 to S1.5 continues until the scheduled list becomes {A, B, C, D, E} and the list of nodes to be scheduled becomes {F, G, H, I, J, K, L}.

[0143] S1.6 completes the hub node:

[0144] After the reverse tracing is complete, the processed hub nodes need to be marked as "completed," indicating that the analysis and processing of that node has been completed. Then, we can continue to perform a forward depth-first search until the entire neural network has been traversed or other termination conditions are met.

[0145] exist Figure 1 In the example, the currently scheduled list is {A,B,C,D,E,F}, and the list to be scheduled is {G,H,I,J,K,L}.

[0146] Steps S1.4 to S1.6 described in S1.7 are repeated cyclically until the list of devices to be scheduled is empty, at which point the network operation ends.

[0147] Throughout the entire processing flow, connectivity is paramount, influencing our selection of nodes and the order in which they are processed. By considering node connectivity, we can more accurately grasp the structure and function of the network, thereby obtaining more accurate and reliable analysis results.

[0148] exist Figure 1In the example, in step S.1.4 (forward depth-first search), hub node J is encountered. In step S.1.5 (backward tracing), the child scheduling list that depends on node G is added to the dependency list of hub node J. The child scheduling list is {H,G,I}. The forward depth-first sorting operation of S1.4 is performed on the child scheduling list. At this time, the scheduled list is {A,B,C,D,E,F,H,G,I}, and the scheduling list is {J,K,L}. In step S.1.6 (complete hub node), the scheduled list is now {A,B,C,D,E,F,H,G,I,J}, and the scheduling list is {K,L}. Step S.1.4 (forward depth-first search) is repeated. Hub node K is encountered. In step S.1.5 (backward tracing), the child scheduling list that depends on node L is added to the dependency list of hub node K. The sub-list to be scheduled is {L}. Perform the forward depth sorting operation S1.4 on the sub-list to be scheduled. At this time, the scheduled list is {A,B,C,D,E,F,H,G,I,J,L}, and the list to be scheduled is {K}. Step S.1.6 (Complete the hub node). At this time, the scheduled list is {A,B,C,D,E,F,H,G,I,J,L,K}, and the list to be scheduled is {}. The sorting ends.

[0149] S2, sorts the allocation priority of intermediate tensors in the computation graph by tensor activity;

[0150] The active length of a tensor indicates that the tensor must exist in memory within a certain period of time, reflecting its active state in memory to a certain extent; that is, the strategy of calculating the offset in memory based on the tensor's activity level can ensure that tensors within similar time periods are as close as possible in memory.

[0151] Furthermore, in step S2, the tensor activity ranking rule is to sort them in a non-increasing manner according to indicators such as tensor activity length, tensor size, and starting point; the earlier the indicator appears, the higher its priority.

[0152] The three metrics in tensor activity ranking—tensor activity length, tensor size, and starting point—play a crucial role in memory management and optimization. The following is a brief introduction to these three metrics:

[0153] 1. Tensor active length

[0154] Definition: Tensor liveness refers to the time period from when a tensor is created in a program until it is no longer used. It reflects the lifespan of a tensor in memory, such as... Figure 3 As shown;

[0155] Importance: Tensor live length is crucial for memory management. A longer live length means that the tensor will occupy memory for a longer period of time, thus increasing the likelihood of memory bottlenecks or fragmentation issues. Prioritizing the allocation and release of tensors with shorter live lengths during sorting helps reduce memory usage and fragmentation, improving memory utilization efficiency.

[0156] 2. Tensor Dimensions

[0157] Definition: Tensor size refers to the amount of storage space occupied by a tensor, which is usually determined by its data type, dimensions, and number of elements;

[0158] Importance: Tensor size is equally important for memory management. Larger tensor sizes mean more memory space is needed for storage, thus making it easier to cause memory shortages or fragmentation problems; when sorting, considering tensor size can help optimize memory allocation, avoiding the mixing of large and small tensors, thereby reducing memory fragmentation;

[0159] 3. Starting point

[0160] Definition: The starting point refers to the creation time or the beginning of the lifecycle of a tensor in a program; Importance: Although the starting point may not be the most direct memory management factor compared to the previous two indicators, it still has some reference value. In some cases, understanding the creation order of tensors can help predict their subsequent memory usage patterns, thus enabling more precise memory management. For example, some algorithms or operations may depend on a specific tensor creation order, and considering the starting point can better meet these needs.

[0161] Tensor active length sorting comprehensively considers these three metrics, enabling more effective memory management and optimization, and improving program performance and stability. During tensor sorting, tensors are first sorted by active length, ensuring that tensors with longer active lengths are processed first. Then, among tensors with the same active length, the tensor with the larger size is processed first. Finally, if the first two metrics are the same, the tensor with the earlier starting point is processed first.

[0162] The tensor active length sorting is assumed to be as shown in the following table:

[0163]

[0164] Tensor activity sorting: 2->3->0->1->5->4->6;

[0165] Because during the tensor activity sorting process, tensors are first sorted according to their active length to ensure that tensors with longer active lengths are processed first. For example, if the active length of Tensor 2 is 3 and the active length of Tensor 3 is 2, then after sorting, Tensor 2 will be before Tensor 3. Then, among tensors with the same active length, tensors with larger sizes are processed first. For example, if the tensor size of Tensor 3 is 5 and the tensor size of Tensor 0 is 4, then after sorting, Tensor 3 will be before Tensor 0.

[0166] Finally, if the first two metrics are the same, ensure that tensors with earlier starting points are processed first.

[0167] In step S2, the tensor activity offset calculation strategy is as follows:

[0168] A large block of memory is pre-allocated, and the intermediate tensor is a data buffer partitioned by offsets within the memory block. This method is called memory offset calculation, and its goal is to minimize the size of the allocated memory block.

[0169] The offset calculation problem can be viewed as a special case of the two-dimensional strip filling problem. This problem can be abstracted and simplified into a filling problem in which a set of rectangular strips with fixed coordinates in a certain dimension enter a container and their size is minimized by adjusting another dimension. If the height of the container represents the temporal domain of memory allocation, then the width of the container represents the memory usage.

[0170] In step S2, the intermediate tensor of the computation graph has the following characteristics:

[0171] The concepts and processes described below will be based on Figure 2 The computation graph is used for representation, and each intermediate tensor in the computation graph has its own lifecycle.

[0172] The lifetime of a tensor is represented by the entire period during which the tensor is active in memory. The lifetime of an intermediate tensor *t* can be defined as {start point, end point}, where the start point and end point are the indices of the producer operator of the intermediate tensor *t* and the last operator that used the intermediate tensor *t* as its input, respectively. These indices come from a topological ordering of the neural network, which is also the execution order of the operators. It is worth noting that no two tensors with intersecting intervals can share memory. The active length represents the length of time the tensor's memory state remains active. Figure 2Tensor 1 has a lifespan of {1,2} and an active length of 1. Tensor 3 has a lifespan of {3,5} and an active length of 2. Since the producer node of tensor 1 is node 1, the lifespan of tensor 1 starts at 1. The consumer node of tensor 1 is node 2, so the lifespan of tensor 1 ends at 2. The active length is the difference between the lifespan ending point and the lifespan starting point.

[0173] S3, find memory gaps, and match the corresponding intermediate tensor when a suitable memory gap is found;

[0174] Memory gaps can be viewed as memory buffers that have been used before a certain point in time and are currently in a freed state;

[0175] For each intermediate tensor, first check the already allocated tensors whose lifetimes intersect with the current tensor to find the smallest memory gap between them, so that the current tensor fits into this gap; if such a gap is found, allocate the current tensor to this gap; otherwise, allocate the current tensor after the bottommost tensor whose lifetime intersects with its current tensor; assign the corresponding offset to the current tensor, and the tensor is now in an allocated state. The method for searching memory gaps is as follows: Figure 4 As shown.

[0176] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A memory offset calculation method based on depth sorting and tensor liveness priority, characterized in that, The method for static memory planning includes the following steps: S1. The node scheduling order is determined based on the depth-first strategy of hub tracing, that is, the node scheduling order is obtained by setting hub nodes and adding tracing function. The tracing function is the process of obtaining the set of ordinary nodes that the hub nodes depend on in reverse and prioritizing the sorting of their dependencies. This process includes two parts: forward sorting and reverse tracing, which complement each other. S2, sorts the allocation priority of intermediate tensors in the computation graph by tensor active length; The active length of a tensor indicates that the tensor must exist in memory within a certain period of time, reflecting its active state in memory to a certain extent; that is, the strategy of calculating the offset in memory based on the tensor's activity level can ensure that tensors within similar time periods are as close as possible in memory. S3, find memory gaps, and match the corresponding intermediate tensor when a suitable memory gap is found; Memory gaps can be viewed as memory buffers that have been used before a certain point in time and are currently in a freed state; For each intermediate tensor, first check the allocated tensors whose lifetimes intersect with the current tensor to find the smallest memory gap between them, so that the current tensor fits the gap; if such a gap is found, the current tensor is allocated to this gap; otherwise, the current tensor is allocated after the bottommost tensor whose lifetime intersects with its current tensor; the corresponding offset is assigned to the current tensor, and the tensor is in the allocated state.

2. The memory offset calculation method based on depth sorting and tensor liveness priority according to claim 1, characterized in that, In step S1, a hub node is defined as a node in the neural network with an in-degree greater than 1, and constant nodes are not considered in the process of counting the in-degree of nodes; First, in a neural network, "in-degree" refers to the number of connections a node receives; for a hub node, its in-degree is greater than 1, which means that multiple other nodes are sending it information or activation signals. Secondly, constant nodes are not considered when calculating the in-degree of a node; constant nodes are those nodes that hold static data, that is, those that always output the same data stream; they are not affected by other nodes; therefore, these constant nodes are excluded when calculating the in-degree of a hub node.

3. The memory offset calculation method based on depth sorting and tensor liveness priority according to claim 1, characterized in that, Step S1 further includes: S1.1 Construct the scheduling list: First, construct the scheduled list and the unscheduled list separately; In this process, the scheduled list refers to the list of nodes or data that have been arranged according to a certain rule or order. The scheduling list contains nodes that have not yet been sorted according to specific rules. In the initial stage of a neural network, a scheduling list is usually generated first, which contains all the nodes that need to be processed. As the processing progresses, some nodes are added to the scheduled list, and the scheduling list is continuously updated until all nodes have been processed. Importing the S1.2 starting node: In this step, the first step is to determine the starting node of the neural network. The starting node is usually the input layer node of the neural network, which is responsible for receiving raw data or features. Importing the starting node means loading these input layer nodes into a sorted list and preparing them for subsequent inference. S1.3 Determine the hub node: Next, we need to determine the hub nodes in the neural network. According to the previously defined definition, hub nodes are those nodes with an in-degree greater than 1, and constant nodes are not considered when calculating the in-degree. Hub nodes can be determined by traversing the entire neural network and calculating the in-degree of each node. During the traversal, constant nodes can be skipped, focusing only on nodes with multiple input connections. S1.4 Forward Depth-First Search: Once the pivot node is identified, a forward depth-first search can begin. Forward depth-first search is a strategy for traversing a neural network. It starts from the initial node and explores each branch as deeply as possible along the network's connections, until the pivot node or leaf node is reached and further exploration is impossible. During the forward depth-first search process, the activation state and calculation results of each node can be recorded to provide a basis for subsequent analysis and processing. Reverse tracing of the S1.5 hub node: During the forward depth-first search process, if a hub node is encountered, it needs to be traced back. Tracing back means finding the node with input connectivity to the hub node from the list of nodes to be scheduled. Through tracing back, we can understand how the hub node is affected by input data or other nodes, as well as their role and scope of influence in the network. S1.6 completes the hub node: After the reverse tracing is completed, the hub nodes that have been processed need to be marked as "completed" to indicate that the analysis and processing of the node have been completed; then, the forward depth-first search can continue to be performed until the entire neural network has been traversed or other termination conditions are met. Steps S1.4 to S1.6 described in S1.7 are repeated until the list of nodes to be scheduled is empty, at which point the network ends. Throughout the entire processing flow, connectivity is maintained and affects the selection of nodes and the processing order.

4. The memory offset calculation method based on depth sorting and tensor liveness priority according to claim 3, characterized in that, Step S1 further includes: S1.1 Construct the scheduling list: Assume that the scheduled list is empty in the initial state, and the list to be scheduled is {A,B,C,D,E,F,G,H,I,J,K,L}; Importing the S1.2 starting node: Suppose that input layer node A is imported into the scheduled list, at which point the scheduled list is {A} and the list to be scheduled is {B,C,D,E,F,G,H,I,J,K,L}. S1.3 Determine the hub node: Suppose that the hub nodes in the neural network are F, J, and K, since their in-degrees are 2, 4, and 2, respectively; S1.4 Forward Depth-First Search: Assume that the forward depth-first search starts from the starting node A, proceeds sequentially along node B, node C, until it encounters the hub node F, at which point the forward process stops; at this point, the scheduled list is {A,B,C}, and the list to be scheduled is {D,E,F,G,H,I,J,K,L}. Reverse tracing of the S1.5 hub node: Suppose that when a hub node F is encountered, it is traced back to its dependencies D and E. Then a sub-scheduling list is created for the hub node's dependencies. The sub-scheduling list is {D,E}. The sub-scheduling list is iterated through S1.2-S1.

5. At this time, the scheduled list is {A,B,C,D,E} and the list to be scheduled is {F,G,H,I,J,K,L}. S1.6 completes the hub node: Suppose that the currently scheduled list is {A,B,C,D,E,F}, and the list to be scheduled is {G,H,I,J,K,L}.

5. The memory offset calculation method based on depth sorting and tensor liveness priority according to claim 4, characterized in that, Step S1.7 further includes: In step S.1.4, during the forward depth-first search, when a hub node J is encountered, in step S.1.5, during the reverse tracing, the node G that it depends on is added to the sub-schedulable list that hub node J depends on. The sub-list to be scheduled is {H,G,I}. Perform the forward depth sorting operation in step S1.4 on the sub-list to be scheduled. At this time, the already scheduled list is {A,B,C,D,E,F,H,G,I}, and the list to be scheduled is {J,K,L}. Step S.1.6 completes the hub node. At this time, the scheduled list is {A,B,C,D,E,F,H,G,I,J}, and the list to be scheduled is {K,L}. In step S.1.4, a forward depth-first search is performed. When a hub node K is encountered, step S.1.5 traces back to the child scheduling list that the hub node K depends on, which is added to the list of children to be scheduled that the hub node K depends on. The sub-list to be scheduled is {L}. Perform the forward depth sorting operation in step S1.4 on the sub-list to be scheduled. At this time, the scheduled list is {A,B,C,D,E,F,H,G,I,J,L}, and the list to be scheduled is {K}. Step S.1.6 completes the hub node. At this point, the scheduled list is {A,B,C,D,E,F,H,G,I,J,L,K}, and the list to be scheduled is {}. The sorting ends.

6. The memory offset calculation method based on depth sorting and tensor liveness priority according to claim 1, characterized in that, In step S2 The rule for sorting tensor activity is to sort them in a non-increasing manner according to indicators such as tensor activity length, tensor size, and starting point. These indicators are ranked in ascending order of priority: The three metrics in tensor activity ranking include: tensor activity length, tensor size, and starting point.

1. Tensor activity length Tensor active length refers to the time period from when a tensor is created in a program until it is no longer used; it reflects the lifespan of a tensor in memory. A longer active length means that tensors need to occupy memory for a longer time, which is more likely to cause memory bottlenecks or fragmentation problems. When sorting, prioritizing the allocation and release of tensors with shorter active lengths helps to reduce memory usage and fragmentation, and improve memory utilization efficiency.

2. Tensor Dimensions Tensor size refers to the amount of storage space occupied by a tensor, which is usually determined by its data type, dimensions, and number of elements; Larger tensor sizes mean more memory space is needed to store them, which makes them more prone to memory shortages or fragmentation. When sorting, considering tensor size can help optimize memory allocation and avoid mixing large and small tensors, thereby reducing memory fragmentation.

3. Starting point The starting point refers to the creation time or the beginning of the lifecycle of a tensor in a program; In some cases, understanding the order in which tensors are created can help predict their subsequent memory usage patterns, thus enabling more precise memory management; this includes the fact that some algorithms or operations may depend on a specific tensor creation order, and considering the starting point can better meet these needs. During tensor sorting, tensors are first sorted according to their active length to ensure that tensors with longer active lengths are processed first. Then, among tensors with the same active length, tensors with larger sizes are processed first. Finally, if the first two criteria are the same, tensors with earlier starting points are processed first.

7. The memory offset calculation method based on depth sorting and tensor start point priority according to claim 6, characterized in that, The tensor active length sorting is assumed to be as shown in the following table: Tensor activity sorting: 2->3->0->1->5->4->6; During the tensor activity sorting process, the tensors are first sorted according to their activity length to ensure that tensors with longer activity lengths are processed first. Assuming that the activity length of Tensor 2 is 3 and the activity length of Tensor 3 is 2, Tensor 2 will be before Tensor 3 after sorting. Then, among tensors with the same active length, ensure that the tensor with the larger tensor size is processed first. For example, if the tensor size of Tensor 3 is 5 and the tensor size of Tensor 0 is 4, then after sorting, Tensor 3 will be before Tensor 0. Finally, if the first two metrics are the same, ensure that tensors with earlier starting points are processed first.

8. The memory offset calculation method based on depth sorting and tensor liveness priority according to claim 1, characterized in that, In step S2, the tensor activity offset calculation strategy is as follows: A large block of memory is pre-allocated, and the intermediate tensor is a data buffer partitioned by offsets within the memory block. This method is called memory offset calculation, and its goal is to minimize the size of the allocated memory block. Offset calculation can be viewed as a special case of a two-dimensional strip filling problem. It can be abstracted and simplified into a filling problem in which a set of rectangular strips with fixed coordinates in a certain dimension enter a container and their size is minimized by adjusting another dimension. If the height of the container represents the temporal domain of memory allocation, then the width of the container represents the memory usage.

9. The memory offset calculation method based on depth sorting and tensor liveness priority according to claim 1, characterized in that, In step S2, the intermediate tensor of the computation graph has the following characteristics: Lifecycle: The lifecycle of a tensor represents the entire process during which the tensor is active in memory; The lifetime of an intermediate tensor t can be defined as {start point, end point}, where the start point and end point are the producer operator of the intermediate tensor t and the index of the last operator that took the intermediate tensor t as its input, respectively. These indices come from a topological sorting of the neural network, which is also the execution order of the operators. It is worth noting that no two tensors with intersecting intervals can share memory.

10. The memory offset calculation method based on depth sorting and tensor liveness priority according to claim 1, characterized in that, In step S3, for an intermediate tensor Ta to be allocated memory: 1) Calculate the required memory size S based on its data type, data layout, dimensions, and other information. a ; 2) Traverse the list of allocated tensors and obtain the set V of tensors whose lifetimes intersect with the intermediate tensor to be allocated. t And sort the tensor set V in ascending order of its starting address. t Sort; 3) Through tensor set V t Retrieve memory gap list V f Compare the size of each memory gap and denote it as f0, f1, ..., f n Find the first tensor T whose size is greater than or equal to its size. a The required memory and the optimal memory gap, where the optimal fit principle is min(f) x -S a )>0; If a suitable memory gap V is found x To satisfy tensor T a The memory requirements are allocated to tensor T. a That is, tensor T a The starting address is V x The starting address; If a sufficiently large memory gap cannot be found, the memory expansion function is used to expand the tensor T. a Assigned to tensor set V t The last tensor T b Following that, namely tensor T a The starting address size is tensor T b The starting address and tensor T b Add up the required memory sizes; 4) Transfer the intermediate tensor T of the memory to be allocated a Add to the list of allocated tensors.

Images