Memory allocation method of multi-level cache neural network accelerator
By performing topological sorting and cache resource allocation optimization on the neural network computation graph, the problem of unreasonable memory allocation in multi-level cache architecture is solved, thereby improving the inference efficiency and data utilization of the neural network.
Patent Information
- Application Number
- CN202511138077.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-14
- Publication Date
- 2025-12-02
AI Technical Summary
In existing technologies, the memory allocation and tensor arrangement of multi-level caching architectures are unreasonable, resulting in low inference efficiency of neural networks.
By reordering the topological order of the neural network computation graph, calculating the lifetime and usage intensity of operator tensors, allocating cache resources according to usage intensity, and optimizing tensor arrangement by combining data arrangement transformation functions, the redundant data handling and incompatible transformations are reduced.
It improves the efficiency of neural network inference, reduces the reuse of cache resources and data conversion time, and achieves efficient neural network computing.
Smart Images
Figure CN121050879A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to neural network accelerators, and more specifically to a memory allocation method for a multi-level cache neural network accelerator. Background Technology
[0002] Since tensor computations in neural network inference typically exhibit spatial and temporal locality, neural network accelerators are generally designed with multi-level caching storage architectures to reuse data as much as possible.
[0003] A well-designed multi-level cache architecture can fully utilize the two localities mentioned above, reducing access to main memory (DRAM). A typical multi-level cache architecture usually consists of three levels: register / local storage (L1_Buffer), on-chip shared cache (L2_Buffer), and off-chip cache (L3_Buffer). Due to limited cache resources, cache usage needs to be carefully planned during neural network inference. Summary of the Invention
[0004] (a) Technical problems to be solved
[0005] To address the aforementioned shortcomings of existing technologies, this invention provides a memory allocation method for a multi-level cache neural network accelerator, which can effectively overcome the defects of low neural network inference efficiency caused by unreasonable memory allocation and tensor arrangement in existing technologies.
[0006] (II) Technical Solution
[0007] To achieve the above objectives, the present invention provides the following technical solution:
[0008] A memory allocation method for a multi-level cached neural network accelerator includes the following steps:
[0009] S1. Reorder the neural network computation graph according to topological order based on the computational dependencies between operators;
[0010] S2. According to the topological order, collect the buffer size required for the input tensor and output tensor of each operator in L2_buffer and L3_buffer respectively;
[0011] S3. According to the topological order, collect the reuse of input tensors during the execution of each operator based on the computational characteristics of each operator;
[0012] S4. Collect information about each operator according to its topological order and the characteristics of each operator.
[0013] S5. Calculate the lifetime of all operator tensors based on the topological order of the operators, and calculate the usage intensity of all operator tensors;
[0014] S6. Sort all operator tensors based on usage intensity, and start allocating from the operator tensor with the highest usage intensity. Allocate operator tensors in the order of L2_buffer, on-chip L3_buffer, and off-chip L3_buffer.
[0015] Preferably, in S5, the lifetime of all operator tensors is calculated based on the topological order of the operators, and the usage intensity of all operator tensors is calculated, including:
[0016] Calculate the lifetime of all operator tensors based on the topological order of the operators;
[0017] Multiply the data size of all operator tensors by the number of reuses to calculate the total data usage of each operator tensor;
[0018] The usage intensity of each operator tensor is calculated by dividing the total data usage of each operator tensor by its lifespan.
[0019] If an operator tensor is used by multiple operators, the total amount of data used is summed, while the usage intensity of operator tensors on L2_buffer and L3_buffer is calculated separately.
[0020] Preferably, in S6, all operator tensors are sorted based on usage intensity, and allocation begins with the operator tensor with the highest usage intensity. The operator tensors are then allocated sequentially in the order of L2_buffer, on-chip L3_buffer, and off-chip L3_buffer, including:
[0021] For some operator tensor X in the allocation:
[0022] S61. Find all operator tensors that overlap with the lifetime of operator tensor X, forming a tensor set T, and all allocated operator tensors, forming a tensor set M. The tensor set M stores the allocation addresses of the allocated operator tensors.
[0023] S62. Traverse tensor sets T and M, and filter out the allocation addresses of all operator tensors located in tensor set T that have been allocated, forming an inaccessible region set U, represented as {(start1, end1), (start2, end2), ..., (start...}. n end n )}, start n end n Here, are the starting address and ending address of the allocation of the nth operator tensor located in the tensor set T and already allocated, respectively, where n is the number of all operator tensors located in the tensor set T and already allocated;
[0024] S63. Obtain the set of accessible regions A in the buffer region to be allocated based on the set of inaccessible regions U, represented in the form {(available_start1, available_end1), (available_start2, available_end2), ..., (available_start1, available_end2), ..., (available_start2 ... m available_end m )}, available_start m available_end m Here, are the starting and ending addresses of the m-th accessible region in the buffer region to be allocated, respectively, where m is the number of all accessible regions in the tensor set T;
[0025] S64. Traverse the set of accessible regions A and allocate buffer regions for the operator tensor X.
[0026] S65. If the operator tensor X is successfully allocated to an available buffer area, the memory allocation table of the corresponding buffer is updated, and subsequent operator tensors are allocated until all operator tensors are allocated to appropriate addresses.
[0027] Preferably, in S64, traversing the accessible region set A and allocating buffer regions for the operator tensor X includes:
[0028] The operator tensor X is allocated a buffer region according to the optimal matching principle:
[0029] Based on the required buffer size Xsize of the operator tensor X in the buffer, the target accessible region that is closest to and not less than the buffer size Xsize is selected from the set of accessible regions A, and this is used as the buffer region allocation result of the operator tensor X.
[0030] Preferably, in S64, traversing the accessible region set A and allocating buffer regions for the operator tensor X includes:
[0031] The operator tensor X is allocated a buffer region according to the nearest matching principle:
[0032] Based on the buffer size Xsize required by the operator tensor X in the buffer, select the target accessible region with the smallest starting address that is not less than the buffer size Xsize from the accessible region set A, and use it as the buffer region allocation result of the operator tensor X.
[0033] Preferably, L2_buffer and L3_buffer each store a memory allocation table that expresses the memory allocation of operator tensors. The horizontal axis of the memory allocation table represents the node number of the topological order, i.e. the lifetime of the operator tensor. The vertical axis of the memory allocation table represents the address representation of L2_buffer / L3_buffer. Each operator tensor occupies the storage space corresponding to its lifetime in a rectangular manner.
[0034] A tensor arrangement method for a multi-level cached neural network accelerator includes the following steps:
[0035] S11. According to the topological order, collect the input tensor arrangement methods supported by each operator in L2_buffer and L3_buffer respectively;
[0036] S12. Based on the characteristics of each operator, collect the data arrangement transformation function from the input tensor to the output tensor of each operator;
[0037] S13. Traverse the operators in topological order, determine the memory allocation location of each operator tensor according to the memory allocation table, and determine the final input tensor arrangement supported by each operator's input tensor according to the memory allocation location.
[0038] S14. Based on the input tensor arrangement supported by the input tensor of each operator and the data arrangement conversion function from the input tensor to the output tensor of each operator, obtain the first output tensor arrangement supported by the output tensor of each operator.
[0039] S15. Find all child node operators that use the output tensor. Traverse the child node operators and take the intersection of the support of all child node operators for the output tensor arrangement method of the output tensor based on the memory allocation position of the output tensor. This will give the second output tensor arrangement method supported by the output tensor of each operator.
[0040] S16. Take the intersection of the first output tensor arrangement and the second output tensor arrangement to determine the final output tensor arrangement supported by the output tensor of each operator.
[0041] S17. Prioritize the final input tensor arrangement and the final output tensor arrangement, and take the first tensor arrangement as the current tensor arrangement result, and the remaining tensor arrangements as candidate tensor arrangement results.
[0042] Preferably, if the final output tensor arrangement supported by the operator's output tensor is empty, then backtrack upwards according to the operator tensor allocation order, return to the previous selectable alternative tensor arrangement result, and re-execute S14~S17.
[0043] For operators that successfully run to the network layer, the tensor arrangement result is assigned to the corresponding operator tensor.
[0044] A multi-level cache architecture for a neural network accelerator includes two levels of cache: an on-core cache and an off-core cache. The on-core cache includes L1_buffer and L2_buffer, and the off-core cache includes on-chip L3_buffer and off-chip L3_buffer.
[0045] L1_buffer, managed by the kernel, is used to store weights and small blocks of input / output activation values;
[0046] L2_buffer has the same structure as L1_buffer and is used to store intermediate results;
[0047] The on-chip L3_buffer is used to store data that cannot be stored in L1_buffer and L2_buffer, as well as other high-frequency data;
[0048] The off-chip L3 buffer, together with the on-chip L3 buffer, constitutes the L3 buffer, which is a large-capacity DDR or HBM used to store all data.
[0049] Preferably, the L1_buffer and L2_buffer are divided into N banks. During neural network inference, the data arrangement of L3_buffer is NCHW or NHWC, which is different from the data arrangement of L1_buffer and L2_buffer.
[0050] The exchange of data between L1_buffer and L3_buffer, and between L2_buffer and L3_buffer, requires hardware intervention and rearrangement.
[0051] (III) Beneficial Effects
[0052] Compared with the prior art, the present invention has the following beneficial effects:
[0053] 1) This invention proposes a memory allocation method for a multi-level cached neural network accelerator. The usage intensity of the operator tensor is calculated based on the number of times the input tensor of convolution or matrix operations is reused and the lifetime of the operator tensor. The intermediate tensors generated during the neural network inference process are allocated based on the ranking of memory data usage intensity. This can maximize data utilization as much as possible when buffer resources are limited, minimize the repeated data transfer between on-chip and off-chip, effectively shorten the neural network inference time, and achieve efficient neural network inference.
[0054] 2) This invention also proposes a tensor arrangement method for a multi-level cached neural network accelerator. Based on the data arrangement method and memory allocation location in the operator tensors L2_buffer / L3_buffer, combined with the characteristics of the operator itself and the data arrangement transformation function, the input tensor and output tensor arrangement results suitable for operator computation are obtained, reducing the data layout re-conversion caused by data arrangement incompatibility and further improving the inference efficiency of the neural network. Attached Figure Description
[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0056] Figure 1 This is a flowchart illustrating the memory allocation method for the multi-level cache neural network accelerator in this invention.
[0057] Figure 2 This is a schematic diagram of the memory allocation table in this invention;
[0058] Figure 3 This is a schematic diagram of the multi-level caching architecture of the neural network accelerator in this invention. Detailed Implementation
[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0060] The following describes the specific process of the memory allocation method for the multi-level cache neural network accelerator provided by this invention (since L1_buffer is managed by the kernel, this method only involves L2_buffer and L3_buffer) with specific examples (e.g. Figure 1 (as shown) and technical effects.
[0061] 1. Reorder the neural network computation graph according to topological order based on the computational dependencies between operators.
[0062] 2. According to the topological order, collect the buffer size required for the input tensor and output tensor of each operator in L2_buffer and L3_buffer respectively.
[0063] Third, according to the topological order, collect the reuse of input tensors during the execution of each operator based on the computational characteristics of each operator.
[0064] Fourth, according to the topological order, collect information about each operator based on its characteristics (such as whether the input or output tensor of a certain operator is not suitable for being placed in the L2_buffer).
[0065] V. Calculate the lifetime of all operator tensors based on the topological order of the operators, and calculate the usage intensity of all operator tensors, including:
[0066] Calculate the lifetime of all operator tensors based on the topological order of the operators;
[0067] Multiply the data size of all operator tensors by the number of reuses to calculate the total data usage of each operator tensor;
[0068] The usage intensity of each operator tensor is calculated by dividing the total data usage of each operator tensor by its lifespan.
[0069] If an operator tensor is used by multiple operators, the total amount of data used is summed, while the usage intensity of operator tensors on L2_buffer and L3_buffer is calculated separately.
[0070] VI. Sort all operator tensors based on usage intensity, and allocate them starting with the operator tensor with the highest usage intensity. The allocation is then performed sequentially according to the order of L2_buffer, on-chip L3_buffer, and off-chip L3_buffer, including:
[0071] For some operator tensor X in the allocation:
[0072] 1) Find all operator tensors whose lifecycles overlap with that of operator tensor X, forming a tensor set T, and all allocated operator tensors, forming a tensor set M. The tensor set M stores the allocation addresses of the allocated operator tensors.
[0073] 2) Traverse tensor sets T and M, and filter out the allocation addresses of all operator tensors located in tensor set T that have been allocated, forming an inaccessible region set U, represented as {(start1, end1), (start2, end2), ..., (start...}. n end n )}, start n end n Here, are the starting address and ending address of the allocation of the nth operator tensor located in the tensor set T and already allocated, respectively, where n is the number of all operator tensors located in the tensor set T and already allocated;
[0074] 3) Obtain the set of accessible regions A in the buffer region to be allocated based on the set of inaccessible regions U, represented as {(available_start1, available_end1), (available_start2, available_end2), ..., (available_start1, available_end2), ..., (available_start2 ... m available_end m )}, available_start m available_end m Here, are the starting and ending addresses of the m-th accessible region in the buffer region to be allocated, respectively, where m is the number of all accessible regions in the tensor set T;
[0075] 4) Traverse the set of accessible regions A and allocate buffer regions for the operator tensor X;
[0076] 5) If operator tensor X is successfully allocated to an available buffer region, update the memory allocation table of the corresponding buffer and continue to allocate subsequent operator tensors until all operator tensors are allocated to appropriate addresses.
[0077] Specifically, 4) Traverse the set of accessible regions A and allocate buffer regions for the operator tensor X, including:
[0078] The operator tensor X is allocated a buffer region according to the optimal matching principle:
[0079] Based on the buffer size Xsize required by the operator tensor X in the buffer, the target accessible region that is closest to and not less than the buffer size Xsize is selected from the accessible region set A, and this is used as the buffer region allocation result of the operator tensor X.
[0080] Alternatively, the operator tensor X can be allocated a buffer region according to the nearest matching principle:
[0081] Based on the buffer size Xsize required by the operator tensor X in the buffer, select the target accessible region with the smallest starting address that is not less than the buffer size Xsize from the accessible region set A, and use it as the buffer region allocation result of the operator tensor X.
[0082] In the technical solution of this application, L2_buffer and L3_buffer respectively store a memory allocation table expressing the memory allocation of operator tensors. The horizontal axis of the memory allocation table represents the node number of the topological order, that is, the lifetime of the operator tensor. The vertical axis of the memory allocation table represents the address representation of L2_buffer / L3_buffer. Each operator tensor occupies the storage space corresponding to its lifetime in a rectangular manner.
[0083] like Figure 2 As shown, the vertical axis of the memory allocation table starts from 0 and ends at 1024, representing a total allocatable address space of 1024. Each operator tensor occupies its corresponding lifetime's storage space in a rectangular manner. For example, operator tensor A occupies the address [0,1024) at node [0,2); operator tensor B occupies the address [512,1024) at node [2,9); operator tensor C occupies the address [0,512) at node [3,5); and tensor D occupies the address [0,512) at node [6,9).
[0084] The above technical solution calculates the usage intensity of operator tensors based on the number of times the input tensors of convolution or matrix operations are reused and the lifetime of operator tensors. It then allocates intermediate tensors generated during neural network inference based on the ranking of memory data usage intensity. This approach maximizes data utilization when buffer resources are limited, minimizes redundant data transfer between on-chip and off-chip environments, effectively shortens neural network inference time, and achieves efficient neural network inference.
[0085] Building upon the memory allocation method for multi-level cached neural network accelerators disclosed above, this application also discloses a tensor arrangement method for multi-level cached neural network accelerators. The specific process and technical effects of the tensor arrangement method for multi-level cached neural network accelerators provided by this invention are described below with specific examples.
[0086] First, according to the topological order, collect the input tensor arrangement methods supported by each operator in L2_buffer and L3_buffer.
[0087] 2. Based on the characteristics of each operator, collect the data arrangement transformation function from the input tensor to the output tensor of each operator.
[0088] 3. Traverse the operators in topological order, determine the memory allocation location of each operator tensor according to the memory allocation table, and determine the final input tensor arrangement supported by each operator's input tensor according to the memory allocation location.
[0089] IV. Based on the input tensor arrangement supported by the input tensor of each operator and the data arrangement conversion function from the input tensor to the output tensor of each operator, the first output tensor arrangement supported by the output tensor of each operator is obtained.
[0090] 5. Find all child node operators that use the output tensor, traverse the child node operators, and take the intersection of the support of all child node operators for the output tensor arrangement method based on the memory allocation position of the output tensor to obtain the second output tensor arrangement method supported by the output tensor of each operator.
[0091] 6. Take the intersection of the first and second output tensor arrangement methods to determine the final output tensor arrangement method supported by the output tensor of each operator.
[0092] 7. Prioritize the final input tensor arrangement and the final output tensor arrangement, and take the first tensor arrangement as the result of this tensor arrangement, and take the remaining tensor arrangement as the candidate tensor arrangement results.
[0093] Specifically, if the final output tensor arrangement supported by the operator's output tensor is empty, then backtrack upwards according to the allocation order of the operator tensors, return to the previous selectable alternative tensor arrangement result, and re-execute steps four to seven above.
[0094] For operators that successfully run to the network layer, the tensor arrangement result is assigned to the corresponding operator tensor.
[0095] The above technical solution, based on the data arrangement and memory allocation location in the operator tensors L2_buffer / L3_buffer, combined with the characteristics of the operator itself and the data arrangement transformation function, obtains the input tensor and output tensor arrangement results suitable for operator computation, reduces the data layout re-conversion caused by data arrangement incompatibility, and further improves the inference efficiency of neural networks.
[0096] Building upon the aforementioned disclosure of memory allocation and tensor arrangement methods for multi-level cache neural network accelerators, this application also discloses a multi-level cache architecture for neural network accelerators. The following section describes the multi-level cache architecture of the neural network accelerator provided by this invention (e.g., [example of specific examples]). Figure 3 (As shown).
[0097] The multi-level cache architecture of the neural network accelerator includes two levels of cache: an on-core cache and an off-core cache. The on-core cache includes L1_buffer and L2_buffer, and the off-core cache includes on-chip L3_buffer and off-chip L3_buffer.
[0098] L1_buffer, managed by the kernel, is used to store weights and small blocks of input / output activation values;
[0099] L2_buffer has the same structure as L1_buffer and is used to store intermediate results;
[0100] The on-chip L3_buffer is used to store data that cannot be stored in L1_buffer and L2_buffer, as well as other high-frequency data;
[0101] The off-chip L3 buffer, together with the on-chip L3 buffer, constitutes the L3 buffer, which is a large-capacity DDR or HBM used to store all data.
[0102] In the technical solution of this application, L1_buffer and L2_buffer are divided into N banks. During neural network inference, the data arrangement of L3_buffer is NCHW or NHWC, which is different from the data arrangement of L1_buffer and L2_buffer.
[0103] The exchange of data between L1_buffer and L3_buffer, and between L2_buffer and L3_buffer, requires hardware intervention and rearrangement.
[0104] To better illustrate the technical effects of the present application's technical solution, a comparative experiment is presented below. Table 1 shows the results of the comparative experiment:
[0105] Table 1 Comparative Experiment Results
[0106]
[0107] As can be seen from Table 1, for typical deep learning models, the FP16 data type can achieve a greater improvement compared to the INT8 data type. This indicates that this approach can maximize data utilization when buffer resources are limited, minimize redundant data transfer between on-chip and off-chip, effectively shorten neural network inference time, and achieve efficient neural network inference.
[0108] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A memory allocation method for a multi-level cache neural network accelerator, characterized in that: Includes the following steps: S1. Reorder the neural network computation graph according to topological order based on the computational dependencies between operators; S2. According to the topological order, collect the buffer size required for the input tensor and output tensor of each operator in L2_buffer and L3_buffer respectively; S3. According to the topological order, collect the reuse of input tensors during the execution of each operator based on the computational characteristics of each operator; S4. Collect information about each operator according to its topological order and the characteristics of each operator. S5. Calculate the lifetime of all operator tensors based on the topological order of the operators, and calculate the usage intensity of all operator tensors; S6. Sort all operator tensors based on usage intensity, and start allocating from the operator tensor with the highest usage intensity. Allocate operator tensors in the order of L2_buffer, on-chip L3_buffer, and off-chip L3_buffer.
2. The memory allocation method for a multi-level cached neural network accelerator according to claim 1, characterized in that: S5 calculates the lifetime of all operator tensors based on their topological order and the usage intensity of all operator tensors, including: Calculate the lifetime of all operator tensors based on the topological order of the operators; Multiply the data size of all operator tensors by the number of reuses to calculate the total data usage of each operator tensor; The usage intensity of each operator tensor is calculated by dividing the total data usage of each operator tensor by its lifespan. If an operator tensor is used by multiple operators, the total amount of data used is summed, while the usage intensity of operator tensors on L2_buffer and L3_buffer is calculated separately.
3. The memory allocation method for a multi-level cached neural network accelerator according to claim 1, characterized in that: In S6, all operator tensors are sorted based on usage intensity, and allocation begins with the operator tensor with the highest usage intensity. The allocation proceeds sequentially according to the L2_buffer, on-chip L3_buffer, and off-chip L3_buffer, including: For some operator tensor X in the allocation: S61. Find all operator tensors that overlap with the lifetime of operator tensor X, forming a tensor set T, and all allocated operator tensors, forming a tensor set M. The tensor set M stores the allocation addresses of the allocated operator tensors. S62. Traverse tensor sets T and M, and filter out the allocation addresses of all operator tensors located in tensor set T that have been allocated, forming an inaccessible region set U, represented as {(start1, end1), (start2, end2), ..., (start...}. n end n )}, start n end n Here, are the starting address and ending address of the allocation of the nth operator tensor located in the tensor set T and already allocated, respectively, where n is the number of all operator tensors located in the tensor set T and already allocated; S63. Obtain the set of accessible regions A in the buffer region to be allocated based on the set of inaccessible regions U, represented in the form {(available_start1, available_end1), (available_start2, available_end2), ..., (available_start1, available_end2), ..., (available_start2 ... m available_end m )}, available_start m available_end m Here, are the starting and ending addresses of the m-th accessible region in the buffer region to be allocated, respectively, where m is the number of all accessible regions in the tensor set T; S64. Traverse the set of accessible regions A and allocate buffer regions for the operator tensor X. S65. If the operator tensor X is successfully allocated to an available buffer area, the memory allocation table of the corresponding buffer is updated, and subsequent operator tensors are allocated until all operator tensors are allocated to appropriate addresses.
4. The memory allocation method for a multi-level cache neural network accelerator according to claim 3, characterized in that: In S64, the accessible region set A is traversed, and buffer regions are allocated for the operator tensor X, including: The operator tensor X is allocated a buffer region according to the optimal matching principle: Based on the required buffer size Xsize of the operator tensor X in the buffer, the target accessible region that is closest to and not less than the buffer size Xsize is selected from the set of accessible regions A, and this is used as the buffer region allocation result of the operator tensor X.
5. The memory allocation method for a multi-level cached neural network accelerator according to claim 3, characterized in that: In S64, the accessible region set A is traversed, and buffer regions are allocated for the operator tensor X, including: The operator tensor X is allocated a buffer region according to the nearest matching principle: Based on the buffer size Xsize required by the operator tensor X in the buffer, select the target accessible region with the smallest starting address that is not less than the buffer size Xsize from the accessible region set A, and use it as the buffer region allocation result of the operator tensor X.
6. The memory allocation method for a multi-level cache neural network accelerator according to claim 3, characterized in that: The L2_buffer and L3_buffer each store a memory allocation table that expresses the memory allocation of operator tensors. The horizontal axis of the memory allocation table represents the node number of the topological order, i.e. the lifetime of the operator tensor. The vertical axis of the memory allocation table represents the address representation of L2_buffer / L3_buffer. Each operator tensor occupies the storage space corresponding to its lifetime in a rectangular manner.
7. A tensor arrangement method for a multi-level cached neural network accelerator, applied to the memory allocation method of the multi-level cached neural network accelerator as described in claim 3, characterized in that: Includes the following steps: S11. According to the topological order, collect the input tensor arrangement methods supported by each operator in L2_buffer and L3_buffer respectively; S12. Based on the characteristics of each operator, collect the data arrangement transformation function from the input tensor to the output tensor of each operator; S13. Traverse the operators in topological order, determine the memory allocation location of each operator tensor according to the memory allocation table, and determine the final input tensor arrangement supported by each operator's input tensor according to the memory allocation location. S14. Based on the input tensor arrangement supported by the input tensor of each operator and the data arrangement conversion function from the input tensor to the output tensor of each operator, obtain the first output tensor arrangement supported by the output tensor of each operator. S15. Find all child node operators that use the output tensor. Traverse the child node operators and take the intersection of the support of all child node operators for the output tensor arrangement method of the output tensor based on the memory allocation position of the output tensor. This will give the second output tensor arrangement method supported by the output tensor of each operator. S16. Take the intersection of the first output tensor arrangement and the second output tensor arrangement to determine the final output tensor arrangement supported by the output tensor of each operator. S17. Prioritize the final input tensor arrangement and the final output tensor arrangement, and take the first tensor arrangement as the current tensor arrangement result, and the remaining tensor arrangements as candidate tensor arrangement results.
8. The tensor arrangement method for a multi-level cached neural network accelerator according to claim 7, characterized in that: If the final output tensor arrangement supported by the operator's output tensor is empty, then backtrack upwards according to the operator tensor allocation order, return to the previous selectable alternative tensor arrangement result, and re-execute S14~S17. For operators that successfully run to the network layer, the tensor arrangement result is assigned to the corresponding operator tensor.
9. A multi-level cache architecture for a neural network accelerator, supporting the memory allocation method for the multi-level cache neural network accelerator as described in claim 3, and the tensor arrangement method for the multi-level cache neural network accelerator as described in claim 7, characterized in that: The system includes a two-level cache consisting of an on-core cache and an off-core cache. The on-core cache includes L1_buffer and L2_buffer, while the off-core cache includes on-chip L3_buffer and off-chip L3_buffer. L1_buffer, managed by the kernel, is used to store weights and small blocks of input / output activation values; L2_buffer has the same structure as L1_buffer and is used to store intermediate results; The on-chip L3_buffer is used to store data that cannot be stored in L1_buffer and L2_buffer, as well as other high-frequency data; The off-chip L3 buffer, together with the on-chip L3 buffer, constitutes the L3 buffer, which is a large-capacity DDR or HBM used to store all data.
10. The multi-level caching architecture of the neural network accelerator according to claim 9, characterized in that: The L1_buffer and L2_buffer are divided into N banks. During neural network inference, the data of L3_buffer is arranged in NCHW or NHWC, which is different from the data arrangement of L1_buffer and L2_buffer. The exchange of data between L1_buffer and L3_buffer, and between L2_buffer and L3_buffer, requires hardware intervention and rearrangement.