On-chip memory tensor allocation method based on address loopback mechanism
By adopting an on-chip memory tensor allocation method based on address loopback mechanism, the problem of wasted storage space in neural network accelerators is solved, achieving more efficient storage space utilization and improved model inference performance.
Patent Information
- Application Number
- CN202511664093.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-13
- Publication Date
- 2026-01-16
AI Technical Summary
In existing technologies, on-chip memory in neural network accelerators wastes storage space during tensor allocation, especially at the boundary between the highest and lowest addresses, where it cannot be effectively utilized, resulting in wasted storage space.
An on-chip memory tensor allocation method based on address loopback mechanism is adopted. By identifying and merging unallocable regions, the address loopback mechanism is used to reallocate storage space, ensuring the continuity between the highest and lowest addresses of the memory and avoiding storage space waste.
This improves the utilization rate of on-chip memory and enhances the inference performance of neural network models.
Smart Images
Figure CN121349906A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to tensor storage of neural network acceleration core, in particular to an on-chip memory tensor allocation method based on address loopback mechanism. BACKGROUND
[0002] In the internal architecture of a neural network accelerator, on-chip memory plays a crucial role, and is one of the core components that determine the performance and efficiency of the accelerator. The core role of on-chip memory can be summarized as relieving the "memory wall" bottleneck and achieving efficient and low-power data supply. Essentially, the on-chip memory of a neural network accelerator is a high-performance and low-power "data transfer station" and "reservoir". It ensures that the "engine" of the computing core can continuously obtain "fuel" (data) through data storage, and is not hindered by the slow "external fuel supply system" (off-chip memory), ultimately achieving high throughput, low latency, and high efficiency of neural network inference computation.
[0003] In the inference process of a neural network model, the unit of computation is a tensor, and the unit of storage space allocation in the on-chip memory is also a tensor. The compiler will plan the location of each tensor used in the on-chip memory before the neural network model inference, and after the computation of the last layer is completed, the computed tensor data will be stored in the corresponding location, and when the next layer needs the tensor data, the data will be read from the location. Therefore, when performing on-chip storage, a continuous address space will be allocated to the same tensor, and the compiler will also perform time and space planning to allow the same address space to be allocated to different tensors at different times during the neural network inference process. As shown in Figure 2 , the compiler of the neural network accelerator will allocate the address space of the tensor in the on-chip memory according to the time of generation and use of each tensor, to ensure that two tensors do not overlap at the same time.
[0004] The above allocation scheme has a disadvantage: the size of the tensor may not completely match the capacity of the on-chip memory, resulting in a certain waste of storage space at the boundary between the highest address and the lowest address of the on-chip memory. As shown in Figure 2 , when computing the 6th to 13th layers of the neural network model, there is free space above and below the address space of tensor 4, but since the addresses of the two free spaces are not continuous, they cannot be allocated to the same tensor. If there is a tensor that needs more space than the free space above and below, it cannot be allocated to the on-chip memory, resulting in a certain waste of storage space. SUMMARY
[0005] (I) Technical problems solved
[0006] In view of the above defects of the prior art, the application provides an on-chip memory tensor allocation method based on an address loopback mechanism, which can effectively overcome the defect of waste of on-chip memory storage space in the prior art.
[0007] (II) Technical solutions
[0008] To achieve the above object, the application is implemented by the following technical solutions:
[0009] The on-chip memory tensor allocation method based on the address loopback mechanism comprises the following steps:
[0010] S1, determining the size and lifetime of all tensors;
[0011] S2, determining a target tensor according to the lifetime, and marking the on-chip memory address space already allocated by the target tensor as an unallocable region;
[0012] S3, merging and sorting the address space marked as the unallocable region in the on-chip memory;
[0013] S4, determining an allocable region according to whether there is space between the highest and lowest addresses of the unallocable region and the on-chip memory;
[0014] S5, allocating the on-chip memory address space for the tensor in the allocable region;
[0015] When the access address of the allocable region is greater than the highest address of the on-chip memory, the address loopback mechanism is used to start accessing from the lowest address of the on-chip memory again.
[0016] Preferably, the step S2 of determining the target tensor according to the lifetime and marking the on-chip memory address space already allocated by the target tensor as the unallocable region comprises:
[0017] finding all target tensors overlapping with the tensor in the lifetime and having allocated the on-chip memory address space, and marking the on-chip memory address space already allocated by the target tensors as the unallocable region.
[0018] Preferably, the lifetime is from the network model layer generating the tensor to the network model layer last using the tensor.
[0019] Preferably, the step S3 of merging and sorting the address space marked as the unallocable region in the on-chip memory comprises:
[0020] for all unallocable regions, if there is space overlap or space continuity, merging is performed;
[0021] for all unallocable regions, sorting is performed according to the starting address from small to large.
[0022] After merging and sorting, all unallocable regions are non-overlapping and their addresses are in ascending order.
[0023] Preferably, in step S4, determining the allocatable region based on whether there is space between the unallocable region and the highest and lowest addresses of the on-chip memory includes:
[0024] If there is space between the last unallocable region, i.e. the unallocable region with the highest address, and the highest address of the on-chip memory, then that space is taken as the first allolocable region and denoted as region A.
[0025] If region A exists, check if there is space between the first allocatable region, i.e. the non-allocatable region with the lowest address, and the lowest address of the on-chip memory. If there is space, then use this space as the second allocatable region and denote it as region B.
[0026] Merge region A and region B into a single, allocatable region. If either region A or region B does not exist, the merge is not performed.
[0027] Preferably, if region A and region B are merged, the starting address of the allocatable region is the highest address of the last unallocable region plus 1, and the ending address of the allocatable region is the starting address plus the sum of the sizes of region A and region B. In this case, the ending address of the allocatable region will be greater than the highest address of the on-chip memory.
[0028] Preferably, the step of restarting access from the lowest address of the on-chip memory when the access address of the allocatable region is greater than the highest address of the on-chip memory, based on the address loopback mechanism, includes:
[0029] When the access address of the allocatable region sent by the computing unit or the upper-level on-chip memory is greater than the highest address of the on-chip memory, the lower bits of the access address of the allocatable region are truncated and used as the actual access address of the on-chip memory. The address loopback mechanism can then be used to start accessing the on-chip memory again from the lowest address.
[0030] The lowest address of the on-chip memory is 0, and the highest address is a power of 2.
[0031] Preferably, the step of truncating the low-order bits of the access address of the allocatable region as the actual access address of the on-chip memory includes:
[0032] If the on-chip memory capacity is 4MB, then the access address of the allocatable region is ANDed with 0x3fffff, the low-order address is extracted, and this low-order address is used as the actual access address of the on-chip memory.
[0033] Preferably, in S5, allocating on-chip memory address space for this tensor within the allocatable region includes:
[0034] Based on a preset allocation strategy, including selecting the first region within the allocable region that meets the size requirement of this tensor, or selecting the region within the allocable region that is closest to the size of this tensor, an on-chip memory address space is allocated for this tensor within the allocable region.
[0035] When allocating the on-chip memory address space of this tensor, the on-chip memory address space allocated to this tensor will not overlap with the on-chip memory address spaces already allocated to other tensors with overlapping lifetimes.
[0036] (III) Beneficial Effects
[0037] Compared with the prior art, the on-chip memory tensor allocation method based on address loopback mechanism provided by this invention, based on the support of on-chip memory for address "loopback" function, and through hardware and software co-design, effectively solves the problem of wasted storage space at the boundary between the highest and lowest addresses of on-chip memory when allocating tensor memory in neural network accelerators, improves the storage space utilization of on-chip memory, and enhances the inference performance of neural network models. Attached Figure Description
[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0039] Figure 1 This is a schematic diagram of the tensor distribution of the on-chip memory in this invention;
[0040] Figure 2 A schematic diagram of the tensor distribution of on-chip memory obtained from the existing allocation scheme. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0042] The following section describes the specific process and technical effects of the on-chip memory tensor allocation method based on the address loopback mechanism provided by this invention, using specific examples.
[0043] S1. Determine the size and lifespan of all tensors.
[0044] S2. Determine the target tensor based on its lifetime, and mark the on-chip memory address space already allocated to the target tensor as an unallocable region, including:
[0045] Find all target tensors that overlap with this tensor in terms of lifetime and have already been allocated on-chip memory address space, and mark the on-chip memory address space of the target tensors that have been allocated as unallocable regions.
[0046] In the technical solution of this application, the lifespan starts from the network model layer that generates the tensor and ends at the network model layer that last uses the tensor.
[0047] S3. Merge and sort the address spaces marked as non-allocatable regions in the on-chip memory, including:
[0048] For all unallocatable regions, if there is spatial overlap or spatial contiguousness, they are merged;
[0049] For all unallocatable regions, sort them in ascending order of their starting addresses;
[0050] After merging and sorting, all unallocable regions are non-overlapping and their addresses are in ascending order.
[0051] S4. Determine the allocatable region based on whether there is space between the unallocable region and the highest and lowest addresses of the on-chip memory, including:
[0052] If there is space between the last unallocable region (i.e., the unallocable region with the highest address) and the highest address of the on-chip memory, then this space is taken as the first allocatable region and denoted as region A (e.g., ...). Figure 1 (as shown)
[0053] If region A exists, then check if there is space between the first allocatable region (i.e., the non-allocatable region with the lowest address) and the lowest address of the on-chip memory. If space exists, then this space is designated as the second allocatable region and denoted as region B (e.g., ...). Figure 1 (as shown)
[0054] Merge region A and region B (this step completes the use of the on-chip memory address loopback mechanism from a software perspective) to form a complete allocatable region. If either region A or region B does not exist, the merge is not performed.
[0055] Specifically, if region A and region B are merged, the starting address of the allocatable region is the highest address of the last unallocable region plus 1, and the ending address of the allocatable region is the starting address plus the sum of the sizes of region A and region B. In this case, the ending address of the allocatable region will be greater than the highest address of the on-chip memory.
[0056] In the technical solution of this application, when the access address of the allocatable region is greater than the highest address of the on-chip memory, the access is restarted from the lowest address of the on-chip memory based on the address loopback mechanism, including:
[0057] When the access address of the allocatable region sent by the computing unit or the upper-level on-chip memory is greater than the highest address of the on-chip memory, the lower bits of the access address of the allocatable region are truncated and used as the actual access address of the on-chip memory. The address loopback mechanism can then be used to start accessing the on-chip memory from the lowest address again (this step completes the use of the on-chip memory address loopback mechanism from a hardware perspective).
[0058] The lowest address of the on-chip memory is 0, and the highest address is a power of 2.
[0059] Specifically, the lower-order bits of the access address of the allocatable region are truncated and used as the actual access address of the on-chip memory, including:
[0060] If the on-chip memory capacity is 4MB, then the access address of the allocatable region is ANDed with 0x3fffff, the low-order address is extracted, and this low-order address is used as the actual access address of the on-chip memory.
[0061] S5. Allocate on-chip memory address space for this tensor within the allocatable region, including:
[0062] Based on a preset allocation strategy, including selecting the first region within the allocable region that meets the size requirement of this tensor, or selecting the region within the allocable region that is closest to the size of this tensor, an on-chip memory address space is allocated for this tensor within the allocable region.
[0063] When allocating the on-chip memory address space of this tensor, the on-chip memory address space allocated to this tensor will not overlap with the on-chip memory address spaces already allocated to other tensors with overlapping lifetimes.
[0064] Figure 1 This is a schematic diagram of the tensor distribution of the on-chip memory in this invention. If there exists a tensor whose size is greater than the size of region A and region B, but less than the sum of the sizes of region A and region B, then this characteristic can be utilized to allocate it to the merged allocatable region, thereby avoiding the waste of storage space at the address boundaries of the on-chip memory.
[0065] likeFigure 1 As shown, when calculating layers 10-13 of the network model, tensor 5 needs to be allocated. The size of this tensor is larger than the size of both region A and region B. According to existing allocation schemes, it cannot be allocated to on-chip memory. However, using the scheme proposed in this invention, tensor 5 can be allocated to the allocatable region formed by merging region A and region B, thus improving the storage space utilization of on-chip memory.
[0066] Using the solution proposed in this invention, tensor allocation is performed on the YOLOv5s deep learning object detection model (input feature map size is 1920*1088, int8 data type) on a 4MB on-chip memory, increasing the tensor allocation by 2040KB (about 2MB) to the on-chip memory, thereby improving the overall inference performance of the model by about 1%.
[0067] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for on-chip memory tensor allocation based on address loopback mechanism, characterized in that: The method comprises the following steps: S1, determining the size and lifetime of all tensors; S2, determining a target tensor according to the lifetime, and marking the on-chip memory address space already allocated by the target tensor as an unallocatable region; S3, merging and sorting the address space in the on-chip memory marked as the unallocatable region; S4, determining an allocatable region according to whether there is space between the highest and lowest addresses of the on-chip memory and the unallocatable region; S5, allocating the on-chip memory address space for the tensor in the allocatable region; When the access address of the allocatable region is greater than the highest address of the on-chip memory, the address is accessed again from the lowest address of the on-chip memory based on an address loop mechanism.
2. The on-chip memory tensor allocation method based on address loopback mechanism according to claim 1, characterized in that: In S2, the target tensor is determined according to the lifetime, and the on-chip memory address space already allocated by the target tensor is marked as an unallocatable region, comprising: All target tensors overlapping with the tensor in the lifetime and having allocated on-chip memory address space are found, and the on-chip memory address space already allocated by the target tensor is marked as an unallocatable region.
3. The on-chip memory tensor allocation method based on address loopback mechanism according to claim 2, characterized in that: The lifetime is from the network model layer where the tensor is generated to the network model layer where the tensor is last used.
4. The on-chip memory tensor allocation method based on address loopback mechanism according to claim 2, characterized in that: In S3, the address space in the on-chip memory marked as the unallocatable region is merged and sorted, comprising: For all unallocatable regions, if there is space overlap or space continuity, merging is performed; For all unallocatable regions, the starting address is sorted from small to large; After merging and sorting, all unallocatable regions do not overlap and the addresses are in ascending order.
5. The on-chip memory tensor allocation method based on address loopback mechanism according to claim 4, characterized in that: In S4, the allocatable region is determined according to whether there is space between the highest and lowest addresses of the on-chip memory and the unallocatable region, comprising: If there is space between the last unallocatable region, i.e., the unallocatable region with the highest address, and the highest address of the on-chip memory, the space is taken as the first allocatable region, and is recorded as region A; If region A exists, it is checked whether there is space between the first allocatable region, i.e., the unallocatable region with the lowest address, and the lowest address of the on-chip memory, if there is space, the space is taken as the second allocatable region, and is recorded as region B; Region A and region B are merged to become a complete allocatable region, if either region A or region B does not exist, merging is not performed.
6. The on-chip memory tensor allocation method based on address loopback mechanism according to claim 5, characterized in that: If region A and region B are merged, the starting address of the allocatable region is the highest address of the last unallocatable region plus 1, and the ending address of the allocatable region is the starting address plus the sum of the sizes of region A and region B, at this time the ending address of the allocatable region will be greater than the highest address of the on-chip memory.
7. The on-chip memory tensor allocation method based on address loopback mechanism according to claim 6, characterized in that: When the access address of the allocatable region is greater than the highest address of the on-chip memory, the address is accessed again from the lowest address of the on-chip memory based on an address loop mechanism, comprising: When the access address of the allocable region sent by the computing unit or the upper-level on-chip memory is greater than the highest address of the on-chip memory, the access address of the allocable region is truncated to the low address as the actual access address of the on-chip memory, that is, the address loop mechanism is used to access the on-chip memory from the lowest address again. The lowest address of the on-chip memory is 0, and the highest address is 2 power.
8. The on-chip memory tensor allocation method based on address loopback mechanism according to claim 7, characterized in that: The truncation of the access address of the allocable region to the low address as the actual access address of the on-chip memory comprises: If the capacity of the on-chip memory is 4MB, the access address of the allocable region is operated once with 0x3fffff, the low address is truncated, and the low address is taken as the actual access address of the on-chip memory.
9. The on-chip memory tensor allocation method based on address loopback mechanism according to claim 5, characterized in that: The allocation of the on-chip memory address space for the tensor in S5 comprises: According to a preset allocation strategy of selecting the first region meeting the size requirement of the tensor from the allocable region or selecting the region closest to the size of the tensor from the allocable region, the on-chip memory address space is allocated for the tensor in the allocable region. When the on-chip memory address space of the tensor is allocated, the on-chip memory address space allocated for the tensor does not overlap with the on-chip memory address space already allocated for the tensor with the remaining survival time.