Memory allocation method of neural network model, readable medium and electronic device

Abstract

The application relates to the field of artificial intelligence technology and discloses a memory allocation method of a neural network model, a readable medium and an electronic device. The memory allocation method of the neural network model comprises the following steps: obtaining a computation graph corresponding to a neural network model and tensor information of tensor data, wherein the tensor information comprises a tensor name, a tensor size, an occupation period and a total memory access footprint; determining a tensor score of each tensor data according to the occupation period and the total memory access footprint of each tensor data and sorting the tensor data according to the tensor score; determining an allocation condition of the tensor data on a cache and a main memory according to a sorting result, and allocating the tensor data on the cache and the main memory according to the allocation condition of the tensor data on the cache and the main memory. The memory allocation method of the neural network model improves the utilization rate of the cache and reduces the occupation and access of the main memory.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a memory allocation method, readable medium, and electronic device for a neural network model. Background Technology

[0002] With the development of artificial intelligence (AI) technology, neural network models are becoming increasingly complex. During the operation of a neural network model, the intermediate data generated by each operator needs to be stored in memory and then read from memory by other operators. Therefore, reasonable memory allocation for intermediate data is necessary.

[0003] Typically, the storage structure of an artificial intelligence system includes main memory DRAM (Dynamic Random Access Memory) and cache units such as SRAM (Static Random-Access Memory), SPM (Scratchpad memory), and TCM (Tightly Coupled Memory).

[0004] However, while main memory has a large storage capacity, it is slow to access, typically with a latency of several hundred time cycles; while cache units have a small storage capacity, usually only a few hundredths of the main memory capacity, but they are much faster to access.

[0005] Therefore, improving cache utilization and reducing main memory usage and access during neural network model runtime is a pressing technical problem that needs to be solved. Summary of the Invention

[0006] The purpose of this application is to provide a memory allocation method, a readable medium, and an electronic device for neural network models.

[0007] The first aspect of this application provides a memory allocation method for a neural network model, comprising: acquiring multiple tensor data to be allocated during the operation of the neural network model; acquiring the total memory access footprint and occupancy period of each tensor data to be allocated during the operation of the neural network model, and determining the allocation order of each tensor data to be allocated based on the total memory access footprint and occupancy period, wherein the earlier the allocation order of the tensor data to be allocated is, the greater the probability that the tensor data to be allocated will be stored in a first memory; and allocating the multiple tensor data to be allocated to the first memory or a second memory respectively based on the allocation order of the multiple tensor data to be allocated, wherein the access speed of the second memory is less than the access speed of the first memory.

[0008] In one possible implementation of the first aspect above, obtaining the total memory access footprint and occupancy period of each tensor data to be allocated during the operation of the neural network model includes: obtaining the tensor size and access count of the tensor data to be allocated; and determining the total memory access footprint of the corresponding tensor data to be allocated based on the product of the access count and the tensor size of the tensor data to be allocated.

[0009] In one possible implementation of the first aspect above, obtaining the total memory access footprint and occupancy period of each tensor data to be allocated during the operation of the neural network model further includes: obtaining the start point and end point of the occupancy period of the tensor data to be allocated; wherein, the start point is used to indicate the first time the tensor data to be allocated is written into memory, and the end point is used to indicate the last time the tensor data to be allocated is read from memory; the difference between the start point and the end point of the occupancy period is the occupancy period of the corresponding tensor data to be allocated.

[0010] In one possible implementation of the first aspect above, determining the allocation order of each tensor data to be allocated based on the total memory access footprint and the occupancy period includes: determining the tensor fraction of each tensor data to be allocated based on the total memory access footprint and the occupancy period; sorting the tensor data according to the size relationship of each tensor fraction; and using the sorting result as the allocation order.

[0011] In one possible implementation of the first aspect above, the tensor fraction of each unallocated tensor data is determined based on the total memory access footprint and occupancy period, including: the unallocated tensor fraction is determined by the following formula:

[0012]

[0013] In the formula, t represents the t-th tensor data to be assigned, and score t Footprint represents the tensor fraction of the t-th tensor data to be assigned. t Size represents the total memory access footprint of the t-th tensor data to be allocated. t Liveness represents the size of the t-th tensor data to be assigned. t .end indicates the end point of the occupancy period of the t-th tensor data to be allocated, Liveness t .start represents the starting point of the occupancy period of the t-th tensor data to be allocated, NumTensors represents the number of tensor data to be allocated, and SRAMSIZE represents the storage capacity of the first memory. Liveness represents the number of accesses to the t-th tensor data to be allocated. t .end-Liveness t .start indicates the occupancy period of the t-th tensor data to be allocated.

[0014] In one possible implementation of the first aspect above, the tensor data is sorted according to the size relationship of each tensor fraction, and the sorting result is used as the allocation order, including: sorting the tensor data to be allocated in descending order according to the corresponding tensor fraction from largest to smallest; and using the descending sorting result as the allocation order.

[0015] In one possible implementation of the first aspect above, allocating multiple tensor data to be allocated to a first memory or a second memory based on the allocation order of multiple tensor data to be allocated includes: taking any tensor data to be allocated that satisfies a first condition as the first tensor data and allocating the first tensor data to the first memory; wherein the first condition is used to indicate that the tensor size of the tensor data to be allocated is less than or equal to the storage capacity of the first memory; and allocating any tensor data to be allocated that does not satisfy the first condition to the second memory.

[0016] In one possible implementation of the first aspect above, after taking any tensor data to be allocated that satisfies the first condition as the first tensor data and allocating the first tensor data to the first memory, the method includes: corresponding to the overlap between the occupancy period of the tensor data to be allocated and the occupancy period of the first tensor data already allocated to the first memory, obtaining a first memory gap in the first memory that satisfies the second condition; wherein the first memory gap is used to represent a continuous unallocated memory space in the first memory; the second condition is used to indicate that the size of the first memory gap is greater than or equal to the tensor size of the tensor data to be allocated; and allocating the tensor data to be allocated to the first memory gap.

[0017] In one possible implementation of the first aspect above, obtaining a first memory gap in the first memory that satisfies the second condition includes: corresponding to the first memory including a plurality of first memory gaps that satisfy the second condition; taking the smallest first memory gap among the plurality of first memory gaps that satisfy the second condition as a second memory gap; and allocating the tensor data to be allocated to the second memory gap.

[0018] In one possible implementation of the first aspect above, obtaining a first memory gap in the first memory that satisfies the second condition includes: corresponding to the absence of a first memory gap in the first memory that satisfies the second condition, allocating the tensor data to be allocated to the second memory.

[0019] In one possible implementation of the first aspect above, after taking any tensor data to be allocated that satisfies the first condition as the first tensor data and allocating the first tensor data to the first memory, the method further includes: obtaining the first location corresponding to the starting address of the memory space of the first memory, corresponding to the fact that the occupancy period of the tensor data to be allocated does not overlap with the occupancy period of the first tensor data already allocated to the first memory; and allocating the tensor data to be allocated to the first location.

[0020] The second aspect of this application provides a readable medium storing instructions that, when executed on an electronic device, cause the electronic device to perform a memory allocation method for any of the neural network models described in the first aspect.

[0021] A third aspect of this application provides an electronic device, including a memory for storing instructions executed by one or more processors of the electronic device, and a processor, one of the processors of the electronic device, for executing a memory allocation method of any of the neural network models described in the first aspect above. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 A schematic diagram of a calculation figure 10 is shown according to some embodiments of this application;

[0024] Figure 2a A schematic diagram of an image recognition scenario is shown according to some embodiments of this application;

[0025] Figure 2b A schematic diagram of an image recognition result is shown according to some embodiments of this application;

[0026] Figure 3 A schematic diagram of another calculation graph is shown according to some embodiments of this application;

[0027] Figure 4 An implementation flowchart of a memory allocation method is shown according to an embodiment of this application;

[0028] Figure 5 A schematic diagram of a memory map is shown according to an embodiment of this application;

[0029] Figure 6 An embodiment of this application illustrates a schematic diagram of the implementation process of a method for allocating tensor data on a cache;

[0030] Figure 7 A schematic diagram of the structure of an electronic device 100 is shown according to some embodiments of this application. Detailed Implementation

[0031] The illustrative embodiments of this application include, but are not limited to, a memory allocation method for a neural network model, a readable medium, and an electronic device.

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described clearly and in detail below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0033] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0034] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0035] Before introducing the technical solutions involved in the embodiments of this application, some of the terms included in the embodiments of this application will be explained.

[0036] (1) Computational Graph

[0037] The term "computation graph" used in the embodiments of this application refers to a way of describing the computation process of a neural network model using a directed graph structure. In practical applications, the basic elements of a directed graph structure are two: nodes and directed edges. Nodes refer to the operators in the computation process of the neural network model; directed edges refer to the directions between different operators, i.e., the dependencies between different operators.

[0038] In some embodiments, a neural network model can be abstracted as tensor data and nodes, and the directed graph structure can be generated based on the tensor data and nodes.

[0039] Figure 1 According to some embodiments of this application, a schematic diagram of calculation Figure 10 is shown. Figure 1 As shown, the computation graph 10 contains 6 nodes and 6 tensor data. Specifically, the 6 nodes are nodes a, b, c, d, e, and f, and the 6 tensor data are tensor data t0, t1, t2, t3, t4, and t5. Taking nodes b, c, and d as examples, for node b, there is a directed edge between node b and node c, pointing from node b to node c. This directed edge indicates that the tensor data t1 generated by node b is the input of node c. At the same time, there is a directed edge between node b and node d, pointing from node b to node d. This directed edge indicates that the tensor data t2 generated by node b is the input of node d.

[0040] (2) Tensor

[0041] The term "tensor" used in the embodiments of this application refers to tensor data, which may include input tensor data, output tensor data in a neural network model, and feature tensor data, etc. Specifically, the tensor data in the embodiments of this application mainly refers to the intermediate data generated by each node.

[0042] In some embodiments, neural network models can be applied to scenarios such as speech recognition and image recognition. For example... Figure 2a In the image recognition scenario shown, the user uses the camera function of mobile phone 100 to capture an image and selects the "object recognition" function. After capturing the image, mobile phone 100 can input the image into a neural network model, which will then perform image recognition and return the recognition result as shown. Figure 2b The results are displayed in window 101.

[0043] In some embodiments, the neural network model is based on input data, such as Figure 2a The image data shown can be used to determine the tensor data and nodes of the input data in the neural network model, and the corresponding computation graph can be generated based on the tensor data and nodes.

[0044] For example, Figure 3 According to some embodiments of this application, a schematic diagram of another computational graph is shown. For example... Figure 3As shown, computation graph 30 includes 7 nodes and 6 tensor data. The 7 nodes are node 0, node 1, node 2, node 3, node 4, node 5, and node 6. The 6 tensor data are tensor data t0, tensor data t1, tensor data t2, tensor data t3, tensor data t4, and tensor data t5. Tensor data t0 has a size of 4 megabytes (MB), tensor data t1 has a size of 1MB, tensor data t2 has a size of 1MB, tensor data t3 has a size of 1MB, tensor data t4 has a size of 1MB, and tensor data t5 has a size of 4MB.

[0045] like Figure 3 As shown, there is a directed edge between node 0 and node 1, pointing from node 0 to node 1, indicating that the tensor data t0 generated by node 0 is the input of node 1. Similarly, there is a directed edge between node 0 and node 5, pointing from node 0 to node 5, indicating that the tensor data t0 generated by node 0 is the input of node 5. There is also a directed edge between node 1 and node 2, pointing from node 1 to node 2, indicating that the tensor data t1 generated by node 1 is the input of node 2. There is a directed edge between node 2 and node 3, pointing from node 2 to node 3, indicating that the tensor data t2 generated by node 2 is the input of node 3. Finally, there is a directed edge between node 3 and node 4, pointing from node 3 to node 4, indicating that the tensor data t3 generated by node 3 is the input of node 4. There is a directed edge between nodes 4 and 5, pointing from node 4 to node 5. This directed edge indicates that the tensor data t4 generated by node 4 is the input to node 5. There is also a directed edge between nodes 5 and 6, pointing from node 5 to node 6. This directed edge indicates that the tensor data t5 generated by node 5 is the input to node 6.

[0046] In some embodiments, tensor data, serving as intermediate data during the operation of a neural network model, is stored in memory. Typically, the storage structure of an artificial intelligence system includes main memory (DRAM) and cache units such as SRAM, SPM, and TCM. However, main memory has a large storage capacity but slow access speed, often with a latency of several hundred time cycles; cache units, while having a smaller storage capacity (typically only a fraction of the main memory's capacity), offer much faster access speeds. Random access storage for tensor data could result in large tensor data being stored in main memory, or tensor data accessed frequently, leading to significant access latency.

[0047] To address the aforementioned technical problems, this application proposes a memory allocation method for use in electronic devices. The method includes: obtaining a computation graph of the neural network model when it calculates the input data, based on input data from a neural network model. This computation graph includes multiple computation nodes and tensor data to be allocated between these nodes (hereinafter referred to as tensor data). Simultaneously, the size, occupancy period, and total memory access footprint of each tensor data are determined. A tensor fraction is determined for each tensor data based on its occupancy period and total memory access footprint, and the tensor data are sorted according to their tensor fractions. Based on the sorting result, the allocation of tensor data in a first memory and a second memory is determined, wherein the access speed of the second memory is lower than that of the first memory (e.g., the first memory is a cache, and the second memory is main memory). Simultaneously, tensor data is allocated between the first and second memories based on their allocation in the first and second memories.

[0048] The tensor data occupancy period refers to the storage period occupied by the tensor data in memory during the computation of the neural network model, for example... Figure 3 As shown, tensor data t0 is used as the output data of node 0 and as the input data of nodes 1 and 5. Therefore, the storage period of tensor data t0 is from node 0 to node 5, meaning the occupancy period of tensor data t0 is <0, 5>, and the length of the occupancy period is 5. The total memory access footprint of tensor data represents the total memory space accessed when tensor data is written to and read from memory during the operation of the neural network model. That is, the total memory access footprint of tensor data is the product of the number of times the tensor data is accessed and the size of the tensor data. For example, as... Figure 3 As shown, the size of tensor data t0 is 4MB. Tensor data t0 is read once by node 1 and node 5 respectively, and written once by node 0. That is, the number of accesses of tensor data t0 is 3, and the memory access footprint of tensor data t0 is 12MB.

[0049] The following uses main memory as an example of the first memory and cache as an example of the second memory to explain some embodiments of this application.

[0050] In some embodiments, a higher tensor score indicates that the tensor data has a shorter occupancy period and a higher access frequency (more accesses), so tensor data with a higher tensor score can be stored in the cache first; a lower tensor score indicates that the tensor data has a longer occupancy period and a lower access frequency (fewer accesses), so when there is not enough memory space in the cache, tensor data with a lower tensor score can be stored in main memory.

[0051] In other embodiments, a lower tensor score indicates that the tensor data has a shorter occupancy period and a higher access frequency, so tensor data with a higher tensor score can be stored in the cache first; a higher tensor score indicates that the tensor data has a longer occupancy period and a lower access frequency, so when there is not enough memory space in the cache, tensor data with a lower tensor score can be stored in main memory.

[0052] In some embodiments, during the runtime of the neural network model, the cache storage capacity may not be sufficient to write all the tensor data. Therefore, memory access footprint analysis and occupancy period analysis are required for the tensor data. The results of the memory access footprint analysis are used as positive feedback, and the results of the occupancy period analysis are used as negative feedback. That is, the tensor score of each tensor data is determined based on the occupancy period of each tensor data and the total memory access footprint of the tensors.

[0053] It is understandable that if tensor data with a long occupancy period is written to the cache as the first tensor data to be written, the tensor data will occupy the memory space in the cache for the entire occupancy period. This part of the memory space cannot be released, which may cause other tensor data to be unable to be written to the cache for a long period of time. Therefore, the result of the occupancy period analysis is used as negative feedback.

[0054] It is understandable that the higher the total memory access footprint of tensor data, the more times the tensor data is accessed. Tensor data with more access frequency is preferentially written to the cache. When the tensor data is accessed (read / written), the latency of accessing from the cache is low and the efficiency is higher. Therefore, the result of memory access footprint analysis is used as positive feedback.

[0055] For example, given two tensors of the same size, if they have the same access period, the tensor with more accesses will have a higher tensor score. As another example, given two tensors of the same size, if they have the same access frequency, the tensor with a longer access period will have a lower tensor score.

[0056] It is understandable that the tensor data is scored based on the results of memory access footprint analysis as positive feedback and the results of occupancy period analysis as negative feedback. Tensor data with high access frequency and short occupancy periods can be prioritized for allocation in the cache. Because these tensor data are accessed frequently, the latency of accessing them from the cache is low. Furthermore, because their occupancy period is short, the memory space they occupy is released after the occupancy period ends, minimizing the impact on other tensor data being written to the cache. Next, tensor data with low access frequency and long occupancy periods are allocated. Because these tensor data are accessed less frequently, even if they cannot be written to the cache, writing them to main memory has a smaller impact on the operation of the neural network model. However, because their occupancy period is long, if they are written to the cache first, they will occupy the cache memory space for the entire occupancy period, potentially preventing other tensor data from being written to the cache. Therefore, the memory allocation method proposed in this application prioritizes allocating tensor data with high tensor scores in the cache and allocates tensor data that cannot be allocated in the cache to main memory, improving cache utilization and reducing main memory usage and access.

[0057] It is understood that the memory allocation method provided in this application can be applied to any electronic device capable of running neural network models, including but not limited to mobile phones, wearable devices (such as smartwatches), tablets, desktops, laptops, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, etc. This application does not impose any restrictions on the specific type of electronic device.

[0058] The following is based on Figure 2a The scene shown and Figure 3 Taking the calculation diagram 30 shown as an example, some technical solutions of this application will be described in detail.

[0059] Figure 4 An implementation flowchart of a memory allocation method is shown according to an embodiment of this application. It can be understood that... Figure 4 The electronic device 100 is the primary entity executing each step of the process shown. For simplicity, the following description... Figure 4 The execution entity for each step in the illustrated process will not be described again. For example... Figure 4 As shown, the process includes, but is not limited to, the following steps:

[0060] S401: Obtain tensor information of the computation graph and tensor data corresponding to the neural network model.

[0061] In some embodiments, firstly, the electronic device 100 acquires the computation graph corresponding to the neural network model, wherein the computation graph includes N nodes and directed edges connecting different nodes, the directed edges of the computation graph carry tensor data, and the computation graph includes M tensor data, where N is an integer greater than 1 and M is a positive integer.

[0062] Secondly, determine the tensor name and tensor size for each tensor data.

[0063] It is understandable that the tensor name can be the number of the tensor data, and the tensor size can be the size of the tensor data.

[0064] Combination Figure 3 The computation graph shown indicates that the tensor names corresponding to tensor data t0 to t5 can be t0, t1, t2, t3, t4, and t5. The size of tensor data t0 is 4MB, the size of tensor data t1 is 1MB, the size of tensor data t2 is 1MB, the size of tensor data t3 is 1MB, the size of tensor data t4 is 1MB, and the size of tensor data t5 is 4MB.

[0065] Then, the occupancy period of each tensor data is calculated.<idx0,idx1> Where idx0 is the starting point of the occupancy period, representing the first node number that uses tensor data as input (i.e., the first time tensor data is written into memory), and idx1 is the ending point of the occupancy period, representing the last node number that uses tensor data as output (i.e., the last time tensor data is read from memory).

[0066] Furthermore, the read and write memory footprints (read memory footprint and write memory footprint) of each tensor data are obtained, and the total memory access footprint of each tensor data is determined based on the read and write memory footprints. That is, the difference between the maximum and minimum memory access addresses during the reading process of each tensor data is obtained as the read memory footprint of each tensor data, and the difference between the maximum and minimum memory access addresses during the writing process of each tensor data is obtained as the write memory footprint of each tensor data; the sum of the read memory footprints and write memory footprints of each tensor data is taken as the total memory access footprint of each tensor data.

[0067] Finally, a list of tensor information is generated based on the tensor name, tensor size, occupancy period, and tensor memory access footprint. For example, refer to Table 1, which illustrates a list of tensor information according to an embodiment of this application.

[0068] Table 1 Tensor Information List

[0069]

[0070] As shown in Table 1, tensor data t0 has a size of 4MB, an occupation period of <0, 5>, a read memory footprint of 8MB, a write memory footprint of 4MB, and a total memory access footprint of 12MB; tensor data t1 has a size of 1MB, an occupation period of <1, 2>, a read memory footprint of 1MB, a write memory footprint of 1MB, and a total memory access footprint of 2MB; tensor data t2 has a size of 1MB, an occupation period of <2, 3>, a read memory footprint of 1MB, a write memory footprint of 1MB, and a total memory access footprint of... 2MB; Tensor data t3 has a size of 1MB, an occupation period of <3,4>, a read memory footprint of 1MB, a write memory footprint of 1MB, and a total memory access footprint of 2MB; Tensor data t4 has a size of 1MB, an occupation period of <4,5>, a read memory footprint of 1MB, a write memory footprint of 1MB, and a total memory access footprint of 2MB; Tensor data t5 has a size of 4MB, an occupation period of <0,5>, a read memory footprint of 4MB, a write memory footprint of 4MB, and a total memory access footprint of 8MB.

[0071] It is understandable that the occupancy periods of tensor data t1, t2, t3, and t4 overlap with the occupancy period of tensor data t0. During the execution of the neural network model, tensor data t0 is written to node 0 (4MB write memory footprint) and read by nodes 1 and 5 (8MB read memory footprint); tensor data t1 is written to node 1 (1MB write memory footprint) and read by node 2 (1MB read memory footprint); tensor data t2 is written to node 2 (1MB write memory footprint) and read by node 3 (1MB read memory footprint); tensor data t3 is written to node 3 (1MB write memory footprint) and read by node 4 (1MB read memory footprint); tensor data t4 is written to node 4 (1MB write memory footprint) and read by node 5 (1MB read memory footprint); tensor data t5 is written to node 5 (4MB write memory footprint) and read by node 6 (4MB read memory footprint).

[0072] S402: Determine the tensor fractions and sort the tensor data according to the tensor fractions.

[0073] In some embodiments, the electronic device 100 obtains the storage capacity of the cache, determines the tensor score of the tensor data based on the obtained cache storage capacity, total memory access footprint, tensor size, and occupancy period, and sorts the tensor data in descending order according to the tensor score.

[0074] For example, tensor fractions can be determined with reference to formula (1).

[0075]

[0076] In the above formula (1), t represents the t-th tensor data, and score t Footprint represents the tensor score of the t-th tensor data. t Size represents the total memory access footprint of the t-th tensor data. t Represents the size of the t-th tensor data, Liveness t .end indicates the end point of the occupancy period of the t-th tensor data, Liveness t .start represents the starting point of the occupancy period of the t-th tensor data, NumTensors represents the total number of tensor data, and SRAMSIZE represents the storage capacity of the cache (SRAM).

[0077] For example, referring to Table 1, assuming the cache storage capacity SRAMSIZE is 10MB, then for tensor data t0, the total memory access footprint is... t0 The size of the tensor data is 12MB. t0 The tensor data is 4MB, and the endpoint of the occupancy period is Liveness. t0 The .end value is 5, representing the starting point of the tensor data's occupancy period (Liveness). t0 With .start set to 0 and the total number of tensor data points (NumTensors) set to 6, the tensor score is obtained. t0 It is 3.47.

[0078] Similarly, the tensor score of tensor data t1 is obtained. t1 The tensor score of tensor data t2 is 1. t2 The tensor score of tensor data t3 is 1. t3 The tensor score of tensor data t4 is 1. t4 The tensor score of tensor data t5 is 1. t5 It is 4.27.

[0079] In some embodiments, when the storage capacity SRAMSIZE of the cache is infinite, all tensor data can be written into the cache when the neural network model is running, without having to write to the main memory. The size of the tensor fraction determined by the above formula (1) is the size of each tensor data. When the tensor data is written into the cache after being sorted from largest to smallest according to the tensor fraction, it can be guaranteed that the tensor data with the largest memory space is written into the cache first, and the memory space occupied by the tensor data written later is smaller than the memory space occupied by the tensor data written earlier. When the occupied memory can be reused, the tensor data written later can be directly written into the memory space already allocated to the tensor data written earlier, without having to allocate new memory space, thus improving the memory reuse rate.

[0080] In other embodiments, when the cache size is not infinite, the storage capacity of the cache may not be able to write all the tensor data when the neural network model is running. It is necessary to perform memory access footprint analysis and occupancy cycle analysis on the tensor data, and use the result of memory access footprint analysis as positive feedback and the result of occupancy cycle analysis as negative feedback.

[0081] Corresponding to formula (1) above, This represents the result of memory access footprint analysis, specifically the number of times the tensor data was accessed; Liveness t .end-Liveness t .start indicates the result of the occupancy period analysis, that is, the period length of the occupancy period.

[0082] It's understandable that if tensor data with long occupancy periods is written to the cache first, it will occupy cache memory space for the entire occupancy period, preventing this memory from being released. This could prevent other tensor data from being written to the cache for an extended period. Therefore, the occupancy period analysis results are used as negative feedback. Conversely, tensor data with more frequent accesses are prioritized for writing to the cache. When this tensor data is accessed (read), accessing it from the cache results in lower latency and higher efficiency. Therefore, the memory access footprint analysis results are used as positive feedback.

[0083] S403: Based on the sorting results of the tensor data, allocate the tensor data in the cache and determine the list of tensors in main memory.

[0084] Electronic device 100 establishes a blank main memory tensor allocation list and a cache tensor allocation list, determines the tensor list according to the sorting result of step S402 above, and traverses the tensor data in the tensor list.

[0085] In some embodiments, if the occupancy period of the current tensor data overlaps with that of the tensor data (first tensor data) in the allocated cache memory space, it is further determined whether there is a memory gap in the cache memory map that is greater than or equal to the size of the current tensor data. If there is a memory gap that is greater than or equal to the size of the current tensor data, the current tensor data is allocated to the starting address of the smallest memory gap (second memory gap) that meets the condition, and the current tensor data is added to the cache tensor allocation list. If there is no memory gap that is greater than or equal to the size of the current tensor data, it is further determined whether the memory space occupied by allocating the current tensor data to the rightmost position of the cache memory map exceeds the storage capacity of the cache. If it does not exceed the capacity, the current tensor data is allocated to the rightmost position of the cache memory map. If it exceeds the capacity, the current tensor data is added to the main memory tensor allocation list.

[0086] The first memory gap includes the rightmost position of the memory map containing memory gaps larger than the current tensor data size and caches larger than or equal to the current tensor data size.

[0087] In other embodiments, if the current tensor data does not overlap with the occupancy period of tensor data in the allocated memory space, it is further determined whether the size of the current tensor data exceeds the storage capacity of the cache; if the size of the current tensor data does not exceed the storage capacity of the cache, the current tensor data is allocated to the first position of the cache (i.e., the leftmost position of the memory map, the beginning of the memory map), and the current tensor data is added to the cache tensor allocation list; if the size of the current tensor data exceeds the storage capacity of the cache, the current tensor data is added to the main memory tensor allocation list.

[0088] For example, Figure 5 A schematic diagram of a memory map is shown according to an embodiment of this application.

[0089] like Figure 5 As shown, a memory map can be represented as a memory space of size SRAMSIZE from offset address 0 to offset address 1 (the cache). Assuming three memory blocks have been allocated for the three tensor data points on this memory map: a memory space of size 36 starting at offset address 0, a memory space of size 64 starting at offset address 64, and a memory space of size 16 starting at offset address 114, then there are two memory gaps on the memory map: gap 1 of size 28 starting at offset address 36 and gap 2 of size 10 starting at offset address 104. The memory map also includes a memory space of size SRAMSIZE from offset address 130 to offset address 1 (the cache), which has not been allocated.

[0090] It can be understood that the leftmost position of the memory map, i.e., the memory space with a starting address and an offset address of 0, and the rightmost position of the memory map, i.e., the memory space with a starting address of the ending address of the last allocated memory space in the memory map + 1, are as follows: Figure 5 The starting address is the memory space with an offset address of 130.

[0091] S404: Allocate tensor data in main memory.

[0092] In some embodiments, the electronic device 100 automatically allocates tensor data in the main memory tensor allocation list.

[0093] It is understood that in other embodiments, the above... Figure 4The steps shown can be combined, deleted, or replaced with other steps that are more conducive to achieving the purpose of this application. For example, steps S403 and S404 can be combined into one step. This application does not impose any restrictions on this.

[0094] To better understand the technical solutions of the embodiments of this application, the following is combined with... Figure 6 This application provides a detailed description of some of its technical solutions.

[0095] Figure 6 An implementation flowchart of a method for allocating tensor data on a cache is shown according to an embodiment of this application. It can be understood that... Figure 6 The electronic device 100 is the primary entity executing each step of the process shown. For simplicity, the following description... Figure 6 The execution entity for each step in the illustrated process will not be described again. For example... Figure 6 As shown, the process includes, but is not limited to, the following steps:

[0096] S601: Sort the tensor data in descending order based on the tensor fractions to obtain a list of tensors.

[0097] In some embodiments, the electronic device 100 sorts the tensor data in descending order according to the tensor fraction, and uses the result of the descending sort as a tensor list.

[0098] In other embodiments, if tensor data with the same tensor fraction exists, the tensor data is sorted according to the tensor data occupancy period. That is, it is sorted in ascending order according to the first node number (idx0) that uses tensor data as input. If the first node number that uses tensor data as input is the same, it is sorted in ascending order according to the last node number (idx1) that uses tensor data as output.

[0099] For example, referring to the tensor data t0 to tensor data t5 shown in Table 1, the tensor score of tensor data t0 is... t0 The tensor score for tensor data t1 is 3.47. t1 The tensor score of tensor data t2 is 1. t2 The tensor score of tensor data t3 is 1. t3 The tensor score of tensor data t4 is 1. t4 The tensor score of tensor data t5 is 1. t5 If the value is 4.27, then sorting the tensor data t0 to t5 in descending order of tensor fraction yields: tensor data t5, tensor data t0, tensor data t1, tensor data t2, tensor data t3, and tensor data t4.

[0100] It can be understood that tensor data t1, tensor data t2, tensor data t3, and tensor data t4 all have a tensor score of 1. The tensor data t1, tensor data t2, tensor data t3, and tensor data t4 are sorted according to their occupancy periods. The occupancy periods of tensor data t1 are <1, 2>, tensor data t2 are <2, 3>, tensor data t3 are <3, 4>, and tensor data t4 are <4, 5>. Sort them in ascending order according to the first node number that uses tensor data as input (i.e., idx0), resulting in: tensor data t1, tensor data t2, tensor data t3, and tensor data t4.

[0101] S602: Create blank cache tensor allocation lists and main memory tensor allocation lists.

[0102] In some embodiments, the electronic device 100 establishes a blank cache (SRAM) tensor allocation list for writing tensor data to the memory space allocated in the cache; similarly, it establishes a blank main memory (DRAM) tensor allocation list for writing tensor data (second tensor data) to the memory space allocated in the main memory.

[0103] S603: Determine whether all tensor data has been allocated.

[0104] In some embodiments, if the determination result is yes, it means that all tensor data in the tensor list has been allocated memory space in the corresponding cache / main memory, then step S615 is executed to end the process.

[0105] In other embodiments, if the determination result is negative, it indicates that there are tensor data in the tensor list that have not yet been allocated corresponding memory space in the cache / main memory, and it is necessary to further determine the allocation status of tensor data in the tensor list. Then, step S604 is executed to obtain the largest tensor data in the tensor list as the current tensor data.

[0106] It is understandable that the determination that all tensor data has been allocated can be made by checking whether the tensor list is an empty set. If the tensor list is an empty set, it means that all tensor data has been allocated. Otherwise, it means that there are still tensor data in the cache / main memory space that have not been allocated. This application does not restrict the method for determining that all tensor data has been allocated.

[0107] S604: Get the first tensor data in the tensor list as the current tensor data.

[0108] In some embodiments, the electronic device 100 obtains the tensor data with the largest tensor score in the tensor list (the first tensor data) as the current tensor data, and further determines the allocation of the current tensor data.

[0109] S605: Determine if the occupancy periods of the current tensor data and the allocated tensor data overlap.

[0110] In some embodiments, if the determination result is yes, it indicates that the occupancy period of the current tensor data and the allocated tensor data overlaps, the memory space of the allocated tensor data cannot be reused, and the current tensor data needs to be allocated new memory space. Then, step S606 is executed to determine whether there is a memory gap in the memory map that is greater than or equal to the size of the current tensor data.

[0111] In other embodiments, if the determination result is negative, it means that the occupancy period of the current tensor data and the allocated tensor data do not overlap, and the memory space of the allocated tensor data can be reused. Then, step S611 is executed to determine whether the size of the current tensor data exceeds the storage capacity of the cache.

[0112] refer to Figure 3 The computation graph shown, taking tensor data t0 as an example, shows that the first node to use tensor t0 as output is node 0, and the last node to use tensor data as input is node 5. That is, idx0 is 0, idx1 is 5, and the occupancy period of tensor data t0 is <0, 5>. It can be understood that the occupancy period <0, 5> of this tensor data overlaps with occupancy periods <0, 1>, <0, 2>, <0, 3>, <0, 4>, <0, 5>, <1, 2>, <1, 3>, <1, 4>, <1, 5>, <2, 3>, <2, 4>, <2, 5>, <3, 4>, <3, 5>, and <4, 5>.

[0113] For example, referring to Table 1, the occupancy period <0, 5> of tensor data t0 overlaps with the occupancy periods <1, 2> of tensor data t1, <2, 3> of tensor data t2, <3, 4> of tensor data t3, and <4, 5> of tensor data t4.

[0114] S606: Determine if there is a memory gap in the memory map that is greater than or equal to the size of the current tensor data.

[0115] In some embodiments, if the determination result is yes, it means that there is a memory gap in the memory map that is greater than or equal to the size of the current tensor data, and the memory space of the memory gap can be allocated to the current tensor data. Then, step S609 is executed to allocate the current tensor data to the starting address of the smallest memory gap that meets the condition.

[0116] In other embodiments, if the determination result is negative, it means that there is no memory gap in the memory map that is greater than or equal to the size of the current tensor data. Therefore, memory space for the current tensor data cannot be allocated. It is necessary to further determine whether there is a sufficiently large memory space on the memory map to store the current tensor data. Then, step S607 is executed to determine whether the memory space occupied by allocating the current tensor data to the rightmost position of the memory map exceeds the storage capacity of the cache.

[0117] For example, refer to Figure 5 A memory map can be represented as a one-dimensional number axis (i.e., the memory axis) containing a memory space of size SRAMSIZE from offset address 0 to the cache offset address. Assuming that three memory blocks have been allocated on this memory map for three tensor data points, with the allocated memory spaces being: a memory space of size 36 starting at offset address 0, a memory space of size 64 starting at offset address 64, and a memory space of size 16 starting at offset address 114, then there are two memory gaps on this memory map: gap 1 of size 28 starting at offset address 36 and gap 2 of size 10 starting at offset address 104. This memory map also includes a memory space of size SRAMSIZE from offset address 130 to the cache offset address; this portion of memory space has not been allocated.

[0118] It can be understood that the leftmost position of the memory map, i.e., the memory space with a starting address and an offset address of 0, and the rightmost position of the memory map, i.e., the memory space with a starting address of the ending address of the last allocated memory space in the memory map + 1, are as follows: Figure 5 The starting address is the memory space with an offset address of 130.

[0119] S607: Determine whether the memory space occupied by allocating the current tensor data to the rightmost position of the memory map exceeds the storage capacity of the cache.

[0120] In some embodiments, if the determination result is negative, it means that after allocating the current tensor data to the rightmost position of the memory map, the memory space occupied does not exceed the storage capacity of the cache, that is, there is enough memory space in the cached memory map to store the current tensor data. Then, step S610 is executed to allocate the current tensor data to the rightmost position of the memory map.

[0121] In other embodiments, if the determination result is yes, it means that after allocating the current tensor data to the rightmost position of the memory map, the memory space occupied exceeds the storage capacity of the cache, that is, there is not enough memory space on the cached memory map to store the current tensor data. Then, step S608 is executed to add the current tensor data to the main memory tensor allocation list and delete the current tensor data from the tensor list.

[0122] S608: Add the current tensor data to the main memory tensor allocation list and remove the current tensor data from the tensor list.

[0123] In some embodiments, after determining that the current tensor data cannot be allocated memory space in the cache, the electronic device 100 adds the current tensor data to the main memory tensor allocation list, and the main memory automatically allocates memory space on the main memory. At the same time, the current tensor data is deleted from the tensor list to facilitate the determination of the allocation status of subsequent tensor data.

[0124] S609: Allocate the current tensor data to the starting address of the smallest memory gap that meets the conditions.

[0125] In some embodiments, after determining that there is memory space available to allocate memory gaps for the current tensor data, the electronic device 100 determines the smallest memory gap on the memory map that is greater than or equal to the size of the current tensor data, uses the starting address of the memory gap as the starting address of the memory space allocated for the current tensor data, and then executes step S613 to add the current tensor data to the cached tensor allocation list and delete the current tensor data from the tensor list.

[0126] S610: Allocate the current tensor data to the rightmost position of the memory map.

[0127] In some embodiments, after the electronic device 100 has a sufficiently large memory space in the cached memory map to store the current tensor data, it allocates the current tensor data to the rightmost position of the memory map, and then executes step S613 to add the current tensor data to the cached tensor allocation list and delete the current tensor data from the tensor list.

[0128] S611: Determine whether the size of the current tensor data exceeds the storage capacity of the cache.

[0129] In some embodiments, if the determination result is yes, it means that the size of the current tensor data exceeds the storage capacity of the cache, and the current tensor data cannot be stored in the cache and needs to be stored in the main memory. Then, step S614 is executed to add the current tensor data to the main memory tensor allocation list and delete the current tensor data from the tensor list.

[0130] In other embodiments, if the determination result is negative, it means that the size of the current tensor data does not exceed the storage capacity of the cache, and the current tensor data can be stored in the cache. Then, step S612 is executed to allocate the current tensor data to the leftmost position (start) of the memory map.

[0131] S612: Allocate the current tensor data to the leftmost position (start) of the memory map.

[0132] In some embodiments, after determining that the current tensor data can be stored in the cache, the electronic device 100 allocates the current tensor data to the leftmost position of the memory map.

[0133] S613: Add the current tensor data to the cached tensor allocation list and remove the current tensor data from the tensor list.

[0134] In some embodiments, after determining the memory space in the current tensor data allocation cache, the electronic device 100 adds the current tensor data to the cache tensor allocation list and deletes the current tensor data from the tensor list to facilitate the determination of the allocation status of subsequent tensor data.

[0135] S614: Add the current tensor data to the main memory tensor allocation list and remove the current tensor data from the tensor list.

[0136] In some embodiments, after determining that the current tensor data cannot be allocated memory space in the cache, the electronic device 100 adds the current tensor data to the main memory tensor allocation list, and the main memory automatically allocates memory space on the main memory. At the same time, the current tensor data is deleted from the tensor list to facilitate the determination of the allocation status of subsequent tensor data.

[0137] S615: End process.

[0138] In some embodiments, the electronic device 100 terminates the process after detecting that all tensor data in the tensor list of the tensor list have been allocated corresponding memory space in the cache / main memory.

[0139] It is understood that in other embodiments, the above... Figure 6 The steps shown can be combined, deleted, or replaced with other steps that are more conducive to achieving the purpose of this application. For example, steps S601 and S602 can be combined into one step. This application does not impose any restrictions on this.

[0140] In summary, the memory allocation method proposed in this application includes: obtaining tensor information of the computation graph and tensor data corresponding to the neural network model, wherein the tensor information includes tensor name, tensor size, occupation period, and total memory access footprint; determining the tensor score of each tensor data based on its occupation period and total memory access footprint, and sorting the tensor data in descending order according to the tensor score; determining the allocation of tensor data in cache and main memory based on the descending order sorting result, and allocating tensor data in cache and main memory according to the allocation of tensor data in cache and main memory. The memory allocation method proposed in this application prioritizes allocating tensor data with large tensor scores in cache, and allocates tensor data that cannot be allocated in cache in main memory, thereby improving cache utilization and reducing main memory occupation and access.

[0141] It is understood that the memory allocation method provided in this application can be applied to any electronic device capable of running neural network models, including but not limited to mobile phones, wearable devices (such as smartwatches), tablets, desktops, laptops, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, etc., and this application does not limit the scope of the application. To facilitate understanding of the technical solutions in this application, the structure of the electronic device to which the memory allocation method provided in this application is applicable is described using electronic device 100 as an example.

[0142] Furthermore, Figure 7 A schematic diagram of the structure of an electronic device 100 is shown according to some embodiments of this application. For example... Figure 7 As shown, the electronic device 100 includes one or more processors 101, system memory 102, non-volatile memory (NVM) 103, communication interface 104, input / output (I / O) devices 105, and system control logic 106 for coupling the processor 101, system memory 102, NVM 103, communication interface 104, and input / output (I / O) devices 105. Wherein:

[0143] The processor 101 can be used to control an electronic device to execute the memory allocation method of this application. The processor 101 may include one or more processing units, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro-programmed control unit (MCU), an AI (Artificial Intelligence) processor, or a field programmable gate array (FPGA). The processing module or processing circuit may include one or more single-core or multi-core processors.

[0144] System memory 102 is volatile memory, such as random-access memory (RAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc. System memory is used for temporary storage of data and / or instructions.

[0145] The non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD), a digital versatile disc (DVD), a solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a Secure Digital (SD) memory card, etc.

[0146] Specifically, system memory 102 and non-volatile memory 103 may each include a temporary copy and a permanent copy of instruction 107. Instruction 107 may include, when executed by processor 101A, causing electronic device 100 to implement the memory allocation methods provided in the embodiments of this application.

[0147] The communication interface 104 may include a transceiver for providing a wired or wireless communication interface for the electronic device 100, thereby enabling communication with any other suitable device via one or more networks. In some embodiments, the communication interface 104 may be integrated into other components of the electronic device 100, for example, the communication interface 104 may be integrated into the processor 101. In some embodiments, the electronic device 100 may communicate with other devices through the communication interface 104; for example, the electronic device 100 may obtain a memory allocation method to be executed from other electronic devices through the communication interface 104.

[0148] Input / output (I / O) device 105 may include input devices such as keyboard, mouse, etc., and output devices such as monitor, etc. Users can interact with electronic device 100 through input / output (I / O) device 105.

[0149] System control logic 106 may include any suitable interface controller to provide any suitable interface to other modules of electronic device 100. For example, in some embodiments, system control logic 106 may include one or more memory controllers to provide an interface to system memory 102 and non-volatile memory 103.

[0150] In some embodiments, at least one of the processors 101 may be packaged together with the logic of one or more controllers for system control logic 106 to form a system in package (SiP). In other embodiments, at least one of the processors 101 may also be integrated on the same chip with the logic of one or more controllers for system control logic 106 to form a SoC.

[0151] It is understood that electronic device 100 can be any electronic device capable of running neural network models, including but not limited to mobile phones, wearable devices (such as smartwatches), tablets, desktops, laptops, handheld computers, and supercomputers.

[0152] The embodiments of this application are not limited to ultra-mobile personal computers (UMPCs), netbooks, cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, etc.

[0153] Understandable. Figure 7The structure of the electronic device 100 shown is merely an example. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0154] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0155] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a Digital Signal Processor (DSP), a microcontroller, an Application Specific Integrated Circuit (ASIC), or a microprocessor.

[0156] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0157] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0158] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0159] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0160] It should be noted that, in the examples and description of this patent, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0161] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A memory allocation method for a neural network model, characterized in that, include: Obtain multiple tensor data to be assigned during the operation of the neural network model; The total memory access footprint and occupancy period of each tensor data to be allocated during the operation of the neural network model are obtained, and the allocation order of each tensor data to be allocated is determined based on the total memory access footprint and the occupancy period. The earlier the allocation order of the tensor data to be allocated is, the greater the probability that the tensor data to be allocated will be stored in the first memory. Based on the allocation order of the multiple tensor data to be allocated, the multiple tensor data to be allocated are respectively allocated to the first memory or the second memory, wherein the access speed of the second memory is less than the access speed of the first memory; The step of determining the allocation order of each of the tensor data to be allocated based on the total memory access footprint and the occupancy period includes: The tensor fraction of each tensor data to be allocated is determined based on the total memory access footprint and the occupancy period. The tensor data is sorted according to the size relationship of each tensor fraction, and the sorting result is used as the allocation order; The tensor fraction is determined by the following formula: ， In the formula, t represents the t-th tensor data to be allocated. Let represent the tensor fraction of the t-th tensor data to be assigned. This represents the total memory access footprint of the t-th tensor data to be allocated. This represents the tensor size of the t-th tensor data to be allocated. This indicates the end point of the occupancy period for the t-th tensor data to be allocated. This indicates the starting point of the occupancy period for the t-th tensor data to be allocated. This indicates the number of tensor data to be allocated. This indicates the storage capacity of the first memory. This represents the number of accesses to the t-th tensor data to be allocated. This represents the occupancy period of the t-th tensor data to be allocated.

2. The memory allocation method according to claim 1, characterized in that, The process of obtaining the total memory access footprint and usage cycle of each of the tensor data to be allocated during the operation of the neural network model includes: Obtain the tensor size and access count of the tensor data to be allocated; The total memory access footprint of the corresponding tensor data to be allocated is determined based on the product of the number of accesses and the tensor size.

3. The memory allocation method according to claim 2, characterized in that, The step of obtaining the total memory access footprint and usage cycle of each of the tensors to be allocated during the operation of the neural network model also includes: Obtain the start and end points of the occupancy period of the tensor data to be allocated; wherein, the start point is used to indicate the first time the tensor data to be allocated is written into memory, and the end point is used to indicate the last time the tensor data to be allocated is read from memory; The occupancy period of the tensor data to be allocated is determined by the difference between the start point and the end point of the occupancy period.

4. The memory allocation method according to claim 1, characterized in that, The step of sorting the tensor data according to the magnitude relationship of each tensor fraction and using the sorting result as the allocation order includes: The tensor data to be assigned are sorted in descending order according to the corresponding tensor fractions; The descending sort result is used as the allocation order.

5. The memory allocation method according to claim 4, characterized in that, The step of allocating the multiple tensor data to be allocated to the first memory or the second memory based on the allocation order of the multiple tensor data to be allocated includes: Any of the tensor data to be allocated that satisfies the first condition is taken as the first tensor data, and the first tensor data is allocated to the first memory; wherein, the first condition is used to indicate that the tensor size of the tensor data to be allocated is less than or equal to the storage capacity of the first memory; Allocate any of the tensor data to be allocated that does not meet the first condition to the second memory.

6. The memory allocation method according to claim 5, characterized in that, After assigning any of the tensor data to be allocated that satisfies the first condition as the first tensor data and allocating the first tensor data to the first memory, the process includes: If the occupancy period of the tensor data to be allocated overlaps with the occupancy period of the first tensor data already allocated to the first memory, a first memory gap in the first memory that satisfies a second condition is obtained; wherein the first memory gap is used to represent a continuous unallocated memory space in the first memory; the second condition is used to indicate that the size of the first memory gap is greater than or equal to the tensor size of the tensor data to be allocated. The tensor data to be allocated is allocated to the first memory gap.

7. The memory allocation method according to claim 6, characterized in that, The step of obtaining the first memory gap in the first memory that satisfies the second condition includes: The first memory includes a plurality of first memory gaps that satisfy the second condition; The smallest first memory gap among multiple first memory gaps that satisfy the second condition is taken as the second memory gap. The tensor data to be allocated is allocated to the second memory gap.

8. The memory allocation method according to claim 6, characterized in that, The step of obtaining the first memory gap in the first memory that satisfies the second condition includes: If there is no first memory gap in the first memory that satisfies the second condition, the tensor data to be allocated is allocated to the second memory.

9. The memory allocation method according to claim 5, characterized in that, After assigning any of the tensor data to be allocated that satisfies the first condition as the first tensor data and allocating the first tensor data to the first memory, the method further includes: Since the occupancy period of the tensor data to be allocated does not overlap with the occupancy period of the first tensor data already allocated to the first memory, the first position corresponding to the starting address of the memory space of the first memory is obtained. The tensor data to be assigned is assigned to the first position.

10. A readable medium, characterized in that, The readable medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the memory allocation method of any one of claims 1-9.

11. An electronic device, characterized in that, include: Memory, used to store instructions executed by one or more processors of an electronic device, and A processor is one of the processors in an electronic device, used to execute the memory allocation method according to any one of claims 1-9.

Images