Npu on-chip storage and weight broadcast architecture and method based on inter-layer data residence

Abstract

The application discloses an NPU on-chip storage and weight broadcast architecture and method based on inter-layer data residence. The architecture comprises a host processor / model compiler, an on-chip configuration and data bus, and a functional area, wherein the functional area comprises an inter-layer residence management area, an on-chip storage area, a data carrying area, a weight broadcast area, an NPU computing array area, and an output and recycling area. The architecture makes the intermediate feature maps that can be reused between network layers be reserved in the on-chip SRAM as much as possible, and makes the weight data be read once through a broadcast network and shared by multiple PEs, so that the number of external storage access times is reduced, power consumption is reduced, and inference throughput and on-chip storage utilization are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an NPU on-chip storage and weighted broadcast architecture and method based on inter-layer data residency. Background Technology

[0002] During neural network inference, there are numerous data dependencies among convolutional layers, matrix multiplication layers, fully connected layers, depthwise separable convolutional layers, normalization layers, activation layers, and residual connection layers. Intermediate feature maps from previous layers are often used as input to subsequent layers. For lightweight neural networks and edge AI models, the amount of data in intermediate feature maps is often greater than the amount of data in the weights; frequent handling of intermediate feature maps leads to higher memory access power consumption and latency.

[0003] A typical NPU architecture usually includes an external memory interface, a DMA controller, on-chip SRAM, a weight cache, an input cache, an output cache, and a PE (processing unit) computation array. In the traditional execution method, before each network layer is executed, the DMA moves the input feature map and weights from external memory to the on-chip cache; after the computation of that layer is completed, the output feature map is written to the output cache, and then written back to external memory by the DMA; when the next layer is executed, the output feature map is read back into the on-chip cache as input. This method is simple to implement, but it results in a large number of unnecessary external memory accesses.

[0004] In edge devices, external storage access power consumption is typically significantly higher than on-chip SRAM access power consumption. This is especially true in battery-powered, low-power AI chips, and energy IoT edge devices, where external DDR, Flash, or off-chip storage access can become a bottleneck for system power consumption and performance. Therefore, reducing intermediate feature map write-backs and rereads, and enabling efficient sharing of weight data among multiple PEs, are key issues in the design of NPU on-chip storage architectures.

[0005] The existing technologies for solving the above problems mainly fall into the following categories:

[0006] (1) NPU storage scheme based on ordinary input cache, weight cache and output cache.

[0007] This scheme divides the on-chip SRAM into input buffer, weight buffer, and output buffer. Inputs and weights are read before each layer is executed, and outputs are written back after each layer is executed. Its advantage is simple control; its disadvantage is the lack of a data residency strategy between layers, resulting in frequent writes of intermediate feature maps back to external memory and high memory access volume.

[0008] (2) NPU data transfer scheme based on double buffering.

[0009] This scheme alternates between loading data and performing calculations using two buffers, allowing data movement and computation to overlap. Its advantage is that it can hide some DMA latency; its disadvantage is that it still reads and writes data hierarchically, failing to fundamentally reduce redundant data movement between layers, and also unable to determine which intermediate data should be retained across layers on-chip.

[0010] (3) Output dwell or weight dwell scheme based on data flow optimization.

[0011] This scheme selects data flow modes such as output-stationary, weight-stationary, or row-stationary within a certain operator, so that certain types of data reside in registers or on-chip caches as much as possible during computation. Its advantage is that it can optimize data reuse within a single layer or a single operator; its disadvantage is that it usually only focuses on the internal structure of a single layer and does not solve the problem of feature map lifecycle management and cross-layer residency between multiple layers.

[0012] (4) Broadcast or multicast scheme based on on-chip network.

[0013] This scheme can send weights or input data from one cache node to multiple compute nodes. Its advantage is reduced redundant reads; its disadvantage is that without coordinated control with inter-layer data residency, weight lifecycle, PE grouping, and cache management, the broadcast network can only function as a normal transmission structure and cannot form a complete NPU (Neural Processing Unit) memory reuse architecture.

[0014] Current NPUs typically execute networks layer by layer, writing the output back to external memory after each layer completes. Since the next layer often immediately uses this output as input, writing it back to external memory and then rereading it results in redundant data transfer. This leads to increased external memory accesses, higher system power consumption, and increased inference latency.

[0015] Existing on-chip caches are typically divided into fixed partitions based on input, weights, and output, without tracking the usage frequency and lifespan of intermediate feature maps in subsequent network layers. Therefore, even if an intermediate feature map will be used quickly in the next layer or subsequent branch, the hardware may still write it back to external memory, resulting in insufficient utilization of the on-chip cache.

[0016] Existing weight loading methods typically involve each computing cluster or each PE group reading weights from the weight cache separately. When multiple PEs use the same weight block for parallel computation, the same weight data is repeatedly read and moved, increasing the pressure on the SRAM read port and the power consumption of on-chip interconnects.

[0017] Current NPUs lack sufficient support for multi-branch networks and residual networks. For an intermediate feature map, if it is to be used by the next layer and then reused in subsequent residual addition, ordinary cache management methods cannot determine its retention time, which can easily lead to premature release or premature write-back.

[0018] Therefore, there is an urgent need for an NPU on-chip storage and weight broadcasting architecture based on inter-layer data residency to solve the following technical problems:

[0019] (1) Solve the problem that after the existing NPU finishes executing each network layer, it writes the intermediate feature map back to the external memory, and then the next layer reads it from the external memory again, resulting in repeated data transfer between layers.

[0020] (2) To solve the problem that in network structures such as multi-layer convolution, pointwise convolution, depthwise convolution, and residual connection, the intermediate feature maps have a short lifespan but are accessed frequently, and the existing architecture cannot effectively reside and reuse them in on-chip cache.

[0021] (3) Solve the problem that the traditional point-to-point reading method causes high pressure on the on-chip SRAM port, data transfer redundancy and increased power consumption when the same weight block needs to be read repeatedly by multiple PE processing units, multiple computing clusters or multiple output channels.

[0022] (4) Solve the problem that the existing NPU on-chip cache is usually only used as a regular input cache, weight cache and output cache, and lacks data lifecycle management, cache space reclamation and cross-layer reuse mechanism based on network layer dependency.

[0023] (5) Solve the problem that the computing array is idle while waiting for weights or intermediate feature maps to be loaded in scenarios with limited external storage bandwidth, which leads to the NPU peak computing power not being fully utilized.

[0024] (6) Solve the problem that existing NPUs cannot determine which intermediate data should be retained, which data should be written back, and which cache space can be released based on the usage of subsequent layers in multi-branch networks, residual networks, and lightweight networks. Summary of the Invention

[0025] The purpose of this application is to provide an NPU on-chip storage and weight broadcasting architecture based on inter-layer data residency, so that reusable intermediate feature maps between network layers are kept in on-chip SRAM as much as possible, and weight data is read once and shared by multiple PEs through the broadcast network, thereby reducing the number of external storage accesses, reducing power consumption, and improving inference throughput and on-chip storage utilization.

[0026] One aspect of this application discloses an NPU on-chip storage and weight broadcasting architecture based on inter-layer data residency, including a main processor / model compiler, an on-chip configuration and data bus, and functional areas, wherein the functional areas include an inter-layer residency management area, an on-chip storage area, a data transport area, a weight broadcasting area, an NPU computing array area, and an output and recycling area;

[0027] The main processor / model compiler is bidirectionally connected to the functional area via the on-chip configuration and data bus;

[0028] The inter-layer residency management area is used to record the data dependencies between different network layers. The inter-layer residency management area includes an inter-layer data residency table, a feature map lifecycle manager, and a cache space allocator. Based on the inter-layer data dependencies generated by model compilation, the inter-layer residency management area uniformly manages cache allocation, data residency determination, and cache release instruction issuance. The inter-layer residency management area outputs cache space scheduling configuration to the on-chip storage area. The inter-layer residency management area issues DMA data transfer control policies to the data transfer area. The inter-layer residency management area provides layer switching timing signals for the weight broadcast area.

[0029] The on-chip storage area includes an on-chip resident cache pool, an input cache window, a weight cache window, and an output cache window; the on-chip resident cache pool is specifically used to store intermediate feature maps reused across layers, and the space of the on-chip resident cache pool is dynamically managed by the cache space allocator; the input cache window caches the input features of the current layer, the weight cache window caches the convolution weights, and the output cache window caches the temporary output results of the array;

[0030] The data transfer area is used to move input data, weight data, and necessary intermediate data from external memory into the on-chip storage area, and to write the final output data or data that cannot be resident back to external memory. The data transfer area includes an external memory interface and a DMA controller. The data transfer area only completes bidirectional data transfer between external memory and the on-chip storage area when the inter-layer resident management area determines that the input data is not resident on the chip or the output data has no reuse value.

[0031] The weight broadcast area includes a weight broadcast controller, a weight broadcast buffer, and a multicast network. The weight broadcast controller reads weight blocks from the weight cache window. After the weight broadcast controller sends the weight blocks into the weight broadcast buffer to complete alignment and fragmentation, the weight broadcast controller distributes the same weight to multiple parallel PE computing clusters in the NPU computing array area through the multicast network.

[0032] The NPU computing array area includes several PE computing clusters. The computing clusters read feature map data from the input buffer window or the on-chip resident buffer pool, receive weight data from the multicast network, perform convolution, matrix multiplication or fully connected computation, and write the computation results back to the output buffer window and transmit them to the output and recycling area.

[0033] The output and recycling area includes an output write-back controller and a cache release controller; the output write-back controller is used to determine the destination of the current layer output data; the cache release controller receives the release instruction issued by the feature map lifecycle manager and reclaims the space occupied by the on-chip resident cache pool after the remaining usage count of the resident feature is cleared to zero.

[0034] According to some embodiments, a portion of SRAM is used as a dynamically allocable on-chip resident cache pool, which dynamically adjusts its cache space based on the network layer execution order and data lifecycle.

[0035] According to some embodiments, the inter-layer data residency table is used to implement inter-layer data residency control; the inter-layer data residency table is generated offline by the model compiler, or written to the NPU by the main processor when the model is loaded; each network layer corresponds to one or more entries in the inter-layer data residency table, and each entry includes the following fields:

[0036] The current layer number is used to identify the network layer that generated the intermediate feature map;

[0037] Output feature map identifier, which is used to identify the intermediate feature map generated in the current layer;

[0038] A list of subsequent layers, which records which subsequent layers will use the feature map;

[0039] Remaining usage count, which records how many more times the feature map needs to be read;

[0040] Retention priority, which indicates whether the feature map is worth keeping in the on-chip cache;

[0041] On-chip cache address, which is used to record the storage location of the feature map in the on-chip resident cache pool;

[0042] The residency size is used to record the cache capacity occupied by the feature map;

[0043] Release conditions, which indicate that the feature map is released immediately after its last use, or after the specified layer ends;

[0044] A write-back flag, which indicates whether the feature map needs to be written back to external memory;

[0045] When executing each layer, the NPU uses the inter-layer data residency table to determine whether the current layer input is already resided in the on-chip cache, whether the current layer output should continue to be retained, and which old feature maps can be released.

[0046] According to some embodiments, the feature map lifecycle manager is used to track the entire process of intermediate feature maps from generation, residence, reuse to release. When a certain network layer completes its computation, the output write-back controller submits the output feature map information to the lifecycle manager. The lifecycle manager queries the inter-layer data residence table to determine whether the feature map will be used by subsequent layers. If the feature map will be used in the next layer or a subsequent short-distance layer, the lifecycle manager marks it as a residence state and requests the cache space allocator to allocate residence cache space for it. If the feature map will not be used again, the lifecycle manager directly releases the feature map or writes it back to external memory. If the feature map needs to be used again by residual branches, jump branches, or multi-branch networks, the lifecycle manager records its remaining usage count and decrements the count after each read. When the remaining usage count is zero, the cache is released.

[0047] According to some embodiments, for different convolution kernels, different output channels, or different matrix blocks, the weight broadcast controller broadcasts sequentially according to the weight block number; for weight blocks used only by a single computing cluster, the weight broadcast controller degenerates into unicast transmission.

[0048] According to some embodiments, the on-chip resident cache pool is controlled by the feature map lifecycle manager; after a single layer ends, the on-chip resident cache pool is not immediately written back or overwritten, but is retained according to the actual dependencies of subsequent network layers; the on-chip resident cache pool adopts a contiguous address allocation method or a block page allocation method.

[0049] According to some embodiments, after each layer of calculation is completed, the output write-back controller determines whether there is a need for subsequent reuse of the array output characteristics based on the inter-layer data residency table;

[0050] If the current layer output will be used immediately by the next layer, the output write-back controller will directly write the current layer output to the resident buffer pool or the next layer input buffer window, without writing it back to external memory.

[0051] If the current layer output will be used by a subsequent branch, but not immediately by the next layer, the output write-back controller writes the current layer output into the resident cache pool and sets the corresponding remaining number of uses.

[0052] If the current layer output will not be used by subsequent layers, but is the final output, the output write-back controller writes the current layer output into the output buffer window and writes the current layer output data back to the external memory through the DMA controller.

[0053] If the current layer output size is too large and the on-chip cache cannot be fully resident, the output write-back controller adopts a block resident strategy, retaining only the tiles (computation tiles) that need to be used in the short term, and writing the rest back to external memory.

[0054] According to some embodiments, the multicast network is used to distribute weight blocks in the weight broadcast buffer to multiple PE computing clusters. The multicast network adopts a bus broadcast method, a tree broadcast method, a cross-connected broadcast method, or an on-chip network multicast method. In the bus broadcast method, the weight broadcast controller places the weight blocks on a shared broadcast bus, and each PE computing cluster receives them according to an enable signal. In the tree broadcast method, the weight data is first sent to several intermediate nodes, and then distributed to multiple computing clusters by the intermediate nodes. In the cross-connected broadcast method or the on-chip network multicast method, the weight broadcast controller sends the weight blocks to the designated computing cluster according to the target computing cluster mask.

[0055] Another aspect of this application provides a method for NPU on-chip storage and weight broadcasting based on inter-layer data residency, comprising the following steps:

[0056] The model compiler or main processor analyzes the inter-layer dependencies of the neural network model and generates an inter-layer data residency table;

[0057] The NPU loads the current layer's input data and weight data; it determines whether the current layer's input features are already stored in the on-chip resident cache pool. If so, it reads the input features directly from the on-chip resident cache pool; otherwise, it starts the DMA controller to load the input features from external memory to the input cache window via the data transfer area.

[0058] After the input data preparation is complete, the current layer weight block is read and stored in the weight cache window;

[0059] The weight broadcast controller extracts the weight blocks in the weight cache window and sends them to the multicast network. The multicast network then synchronously distributes the weights to all PE computing clusters.

[0060] Each PE computation cluster combines input features and broadcast weights to complete convolutional inference computation in parallel;

[0061] After the operation is completed, the output characteristics are written to the output buffer window;

[0062] After the current layer completes the calculation, the output write-back controller determines whether the output feature needs to be used by subsequent layers. If it needs to be used by subsequent layers, it is written into the on-chip resident cache pool, and the lifecycle manager registers the number of times the output feature is used. If it does not need to be used by subsequent layers, the output feature is directly written back to external memory through the DMA controller or the cache release controller releases the on-chip cache space occupied by the output feature.

[0063] When a subsequent layer reads an output feature residing in the on-chip resident cache pool, the lifecycle manager decrements the remaining usage count of the output feature by one; if the remaining usage count is zero, the cache release controller releases the on-chip cache space occupied by the output feature.

[0064] The technical solution disclosed in this application records the subsequent use layer, usage count, residence time, and cache location of intermediate feature maps through an inter-layer data residence table, ensuring that reusable data is retained in on-chip SRAM as much as possible; it unifies the management of input cache, output cache, and intermediate feature cache through an on-chip residence cache pool, reducing space waste caused by fixed cache partitioning; it automatically releases cache space after the intermediate feature map is last used through a feature map lifecycle manager, improving on-chip SRAM utilization; it enables the same weight block to be distributed to multiple PE computing clusters simultaneously after a single read through a weight broadcast controller and an on-chip broadcast network, reducing redundant reads; it coordinates residence policies with DMA transfer control, ensuring that only data that cannot reside on-chip or will not be used for a long time is written back to external memory, thereby reducing external bandwidth consumption; and it enables the NPU computing array to minimize idle periods while waiting for data through the collaborative work of the computing scheduler, cache manager, and weight broadcast network, improving inference throughput. Through the inter-layer data residence table and feature map lifecycle manager, the NPU can know whether each intermediate feature map will be used again, how many times it will be used, which layer it should be retained in, and when it should be released. Therefore, the output of the previous layer does not need to be written back to external memory immediately after each layer ends. Instead, it can reside and be reused across layers in the on-chip cache, thereby significantly reducing the number of external memory accesses. Through the on-chip resident cache pool and cache space allocator, the static usage of on-chip SRAM is improved from a fixed input and output cache to a dynamic usage based on data lifecycle. Therefore, on-chip storage space can be prioritized for retaining intermediate feature maps with high access value that are quickly reused or reused multiple times, improving on-chip cache utilization. The technical solution disclosed in this application, through a weight broadcast controller and a multi-broadcast network, allows the same weight block to be read once and used by multiple PE computing clusters, avoiding repeated access to the weight cache by multiple computing clusters. This reduces the pressure on the on-chip SRAM read port, reduces data transfer power consumption, and reduces the idle time of the computing array waiting for weight loading.

[0065] Compared with the traditional layer-by-layer read-write NPU storage architecture, the technical solution disclosed in this application reduces the number of intermediate feature map write-backs and re-reads into external memory; compared with the single-layer data flow optimization scheme, the technical solution disclosed in this application further extends to data lifecycle management between network layers; compared with the ordinary on-chip broadcast structure, the technical solution disclosed in this application combines weighted broadcasting with inter-layer resident cache management to form a complete on-chip storage reuse architecture. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This diagram illustrates an NPU on-chip storage and weight broadcasting architecture based on inter-layer data residency, according to an example embodiment.

[0068] Figure 2 This diagram illustrates the core architecture of inter-layer data residency and weight broadcasting according to an example embodiment.

[0069] Figure 3 This diagram illustrates a workflow for inter-layer data residency and weight broadcasting according to an example embodiment.

[0070] Figure 4 A timing diagram of interlayer feature map dwell time is shown according to an example embodiment. Detailed Implementation

[0071] The embodiments of this application will now be described in detail with reference to the accompanying drawings. It should be understood that the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0072] Those skilled in the art should understand that the following specific embodiments or implementation methods are a series of optimized configurations listed in this application to further explain the specific application content. These configuration methods can be combined or used in conjunction with each other, unless this application explicitly states that some or a specific embodiment or implementation method cannot be associated with or used in conjunction with other embodiments or implementation methods. Furthermore, the following specific embodiments or implementation methods are only considered as optimized configurations and are not intended to limit the scope of protection of this application.

[0073] Example 1:

[0074] Figure 1 This diagram illustrates an NPU on-chip storage and weight broadcasting architecture based on inter-layer data residency, according to an example embodiment.

[0075] like Figure 1 As shown, an NPU on-chip storage and weight broadcasting architecture based on inter-layer data residency includes a main processor / model compiler 101, an on-chip configuration and data bus 102, and functional areas. The functional areas include an inter-layer residency management area 103, an on-chip storage area 104, a data transport area 105, a weight broadcasting area 106, an NPU computing array area 107, and an output and recycling area 108.

[0076] The main processor / model compiler 101 is bidirectionally connected to the functional area via on-chip configuration and data bus 102.

[0077] The inter-layer residency management area 103 is used to record the data dependencies between different network layers. Based on the inter-layer data residency table generated by the model compiler or main processor, it determines whether each intermediate feature map needs to be kept in the on-chip cache, to which layer it is kept, which subsequent layers it is used by, and when it is released.

[0078] The inter-layer residency management area 103 includes an inter-layer data residency table, a feature map lifecycle manager, and a cache space allocator. Based on the inter-layer data dependencies generated during model compilation, the inter-layer residency management area 103 uniformly manages cache allocation, data residency determination, and cache release instruction issuance. It outputs cache space scheduling configurations to the on-chip storage area 104, issues DMA data transfer control policies to the data transfer area 105, and provides layer switching timing signals to the weight broadcast area 106.

[0079] The on-chip storage area 104 includes an on-chip resident cache pool, an input cache window, a weight cache window, and an output cache window. In this embodiment, a portion of SRAM is used as a dynamically allocable on-chip resident cache pool, and the cache space is dynamically adjusted according to the network layer execution order and data lifecycle. The on-chip resident cache pool is specifically used to store intermediate feature maps reused across layers, and its space is dynamically managed by a cache space allocator. The input cache window caches the input features of the current layer, the weight cache window caches the convolution weights, and the output cache window caches the temporary output results of the array.

[0080] The data transfer area 105 is used to move input data, weight data, and necessary intermediate data from external memory to on-chip storage area 104, and to write the final output data or data that cannot be resident back to external memory. The data transfer area 105 includes an external memory interface and a DMA controller. The data transfer area 105 only completes bidirectional data transfer between external memory and on-chip storage area 104 when the inter-layer resident management area 103 determines that the input data is not resident on-chip or the output data has no reuse value.

[0081] The weight broadcast area 106 is used to read a weight block from the weight cache once and distribute it simultaneously to multiple PE computing clusters, avoiding multiple computing clusters reading the same weight repeatedly. The weight broadcast area 106 includes a weight broadcast controller, a weight broadcast buffer, and a multicast network. The weight broadcast controller reads a weight block from the weight cache window, sends the weight block to the weight broadcast buffer for alignment and fragmentation, and then distributes the same weight synchronously to multiple parallel PE computing clusters within the NPU computing array area 107 via the multicast network.

[0082] The NPU computing array area 107 includes several PE computing clusters. Each computing cluster reads feature map data from the input buffer window or the on-chip resident buffer pool, receives weight data from the multicast network, performs convolution, matrix multiplication or fully connected computation, writes the operation results back to the output buffer window and transmits them to the output and recycling area 108.

[0083] The output and recycling area 108 determines whether the current layer's output should continue to reside on-chip, be written to the next layer's input window, or be written back to external memory based on the inter-layer data residency table. When a residency feature map completes its last use, its cache space is released. The output and recycling area 108 includes an output write-back controller and a cache release controller. The output write-back controller determines the destination of the current layer's output data, while the cache release controller receives release commands from the feature map lifecycle manager and reclaims the space occupied by the on-chip residency cache pool after the remaining usage count of the residency feature is cleared.

[0084] The inter-layer data residency table is the core control structure for implementing inter-layer data residency. This table can be generated offline by the model compiler or written to the NPU by the main processor during model loading. Each network layer corresponds to one or more entries, and each entry includes at least the following fields:

[0085] The current layer number is used to identify the network layer that generated this intermediate feature map.

[0086] Output feature map identifiers, used to identify intermediate feature maps generated in the current layer.

[0087] The subsequent layer list records which subsequent layers will use this feature map.

[0088] Remaining usage count is used to record how many more times the feature map needs to be read.

[0089] Retention priority indicates whether the feature map is worth keeping in the on-chip cache.

[0090] The on-chip cache address is used to record the storage location of the feature map in the on-chip resident cache pool.

[0091] Resident size, used to record the cache capacity occupied by this feature map.

[0092] Release conditions are used to indicate whether the feature map is released immediately after its last use or after the specified layer ends.

[0093] The write-back flag indicates whether the feature map needs to be written back to external memory.

[0094] This table allows the NPU to determine, when executing each layer, whether the current layer's input is already in the on-chip buffer, whether the current layer's output should be retained, and which old feature maps can be released.

[0095] The feature map lifecycle manager tracks the entire process of intermediate feature maps from generation, residence, reuse to release. When a network layer completes computation, the output write-back controller submits the output feature map information to the lifecycle manager. The lifecycle manager then queries the inter-layer data residence table to determine whether the feature map will be used by subsequent layers.

[0096] If the feature map will continue to be used in the next layer or subsequent short-range layers, the lifecycle manager marks it as resident and requests the cache space allocator to allocate resident cache space for it. If the feature map will no longer be used, it is directly released or written back to external memory. If the feature map needs to be reused by residual branches, jump branches, or multi-branch networks, the lifecycle manager records its remaining usage count and decrements the count after each read. When the remaining usage count is zero, cache release is triggered.

[0097] The cache space allocator is used to manage the on-chip resident cache pool. Traditional NPUs typically divide the on-chip SRAM into fixed input cache, weight cache, and output cache. However, in this embodiment, a portion of the on-chip SRAM is used as a dynamic resident cache pool, which is allocated by the cache space allocator according to feature map size, resident priority, and subsequent usage distance.

[0098] When on-chip cache space is sufficient, intermediate feature maps that will be used by the next layer or used multiple times are prioritized for retention. When on-chip cache space is insufficient, the cache space allocator selects to release data of lower value based on resident priority, such as data that will be used later but is far away, data that is used only once and has low external reread cost, or data that has been used for the last time.

[0099] A weight broadcast controller addresses the issue of the same weight block being repeatedly read by multiple PE computation clusters. In convolution, matrix multiplication, or fully connected computations, the same weight block may need to be used simultaneously by multiple output locations, multiple input channel blocks, or multiple PE computation clusters. In traditional solutions, each computation cluster accesses the weight buffer separately, resulting in repeated reads. In this embodiment, the weight broadcast controller reads a weight block from the weight buffer window and writes it to the weight broadcast buffer based on the current layer operator type, PE grouping method, and weight reuse range. Subsequently, the multicast network simultaneously sends the weight block to multiple PE computation clusters. After receiving the same weight block, each PE computation cluster performs multiply-add operations with its respective input feature block. For different convolution kernels, different output channels, or different matrix blocks, the weight broadcast controller can broadcast sequentially according to the weight block number. For weight blocks used only by a single computation cluster, the weight broadcast controller can also degenerate into unicast transmission to avoid unnecessary broadcast power consumption.

[0100] Figure 2 This diagram illustrates the core architecture of inter-layer data residency and weight broadcasting according to an example embodiment.

[0101] like Figure 2As shown, the complete data and control signal links are divided according to the weight broadcast area 106, the inter-layer residency management area 103, the on-chip storage area 104, and the NPU computing array area 107. The inter-layer residency management area 103 is composed of an inter-layer data residency table, a feature map lifecycle manager, and a cache space allocator connected in sequence. The inter-layer data residency table inputs rules such as neural network inter-layer feature reuse, residency duration, and usage count to the feature map lifecycle manager. The feature map lifecycle manager records the remaining read counts of each intermediate feature map in the residency cache pool in real time and synchronously transmits the count changes to the cache space allocator. The cache space allocator is connected downwards to the horizontally arranged weight cache window, input cache window, output cache window, and residency cache pool at the bottom, dynamically managing the storage space allocation, reclamation, and read / write permissions of the four types of cache windows, while simultaneously sending network layer switching timing and weight loading scheduling related control signals upwards to the weight broadcast controller. The weight broadcast area 106 includes a weight broadcast controller, a weight broadcast buffer, and a multi-channel broadcast network connected in series. The weight broadcast controller reads the current layer weight block data from the weight cache window, performs data fragmentation and timing alignment, and stores it in the weight broadcast buffer. Then, the multi-channel broadcast network synchronously distributes the same set of weight data to multiple sets of parallel PE computing clusters in the NPU computing array area 107 through multi-branch parallel links. Each set of PE computing clusters has two independent data input channels. One channel receives the shared weights sent by the multi-channel broadcast network, and the other channel reads the current layer input features stored in the input cache window. After all PE computing clusters complete the inference operation in parallel, they uniformly summarize the output features to the output cache window. The output cache window can push the operation results to the resident cache pool to complete cross-layer resident storage. The reused intermediate feature maps in the resident cache pool can be directly sent back to the input cache window for use by subsequent network layers without needing to be moved through external memory.

[0102] Interlayer residency management area 103 coordinates cache resource scheduling, weight broadcasting path enables weight sharing among multiple PE clusters, on-chip cache window completes feature data residency reuse, and PE computing array performs computing power calculations in parallel. Weight broadcasting area 106 and interlayer residency management area 103 work together to reduce data transport overhead.

[0103] The on-chip resident cache pool is used to store intermediate feature maps reused across layers. Unlike ordinary output caches, this cache pool is controlled by the feature map lifecycle manager and is not immediately written back or overwritten after a single layer ends; instead, it is retained based on the actual dependencies of subsequent network layers. The on-chip resident cache pool can use either contiguous address allocation or block-based paging allocation. Contiguous address allocation is suitable for scenarios with large intermediate feature map sizes and continuous access; block-based paging allocation is suitable for scenarios with multi-branch networks and mixed resident feature maps of different sizes. To avoid cache fragmentation, the cache space allocator can maintain a free block table and an occupied block table. When a feature map is released, its occupied space is added to the free block table. If subsequent feature maps require a large contiguous space, the cache space allocator can perform block merging or choose to store in blocks.

[0104] The output write-back controller determines the destination of the current layer's output data. After each layer's calculation is complete, the output write-back controller makes a determination based on the inter-layer data residency table.

[0105] If the output of the current layer will be used immediately by the next layer, it is written directly to the resident buffer pool or the input buffer window of the next layer, without being written back to external memory.

[0106] If the output of the current layer will be used by subsequent branches, but not immediately by the next layer, it is written to the resident cache pool and the corresponding remaining number of uses is set.

[0107] If the current layer output will not be used by subsequent layers, but is the final output, it is written to the output buffer window and then written back to external memory via DMA.

[0108] If the current layer output size is too large and the on-chip cache cannot hold all of it, a block-based resident strategy can be adopted, retaining only the tiles that will be needed in the short term and writing the rest back to external memory.

[0109] A multi-path weight broadcast network is used to distribute weight blocks in the weight broadcast buffer to multiple PE compute clusters. This network can be implemented using bus broadcast, tree broadcast, cross-switch broadcast, or on-chip network multicast. In bus broadcast mode, the weight broadcast controller places the weight blocks on a shared broadcast bus, and each PE compute cluster receives them based on an enable signal. This method has a simple structure and is suitable for NPUs with a small number of compute clusters. In tree broadcast mode, weight data is first sent to several intermediate nodes, which then distribute it to multiple compute clusters. This method reduces the load on a single bus and is suitable for NPUs with a large number of compute clusters. In cross-switch or on-chip network multicast mode, the weight broadcast controller sends the weight blocks to the designated compute cluster based on the target compute cluster mask. This method offers greater flexibility and is suitable for multi-tasking or multi-partition NPUs.

[0110] Example 2:

[0111] Figure 3 This diagram illustrates a workflow for inter-layer data residency and weight broadcasting according to an example embodiment.

[0112] like Figure 3 As shown, an NPU on-chip storage and weight broadcasting method based on inter-layer data residency includes the following steps:

[0113] S301, a model compiler that parses offline the data dependencies of all network layers of a neural network, as well as the reuse scenarios and number of times intermediate feature maps are reused;

[0114] S302, Generate an inter-layer data residency table based on the parsing results, including record features, subsequent usage layers, remaining usage times, residency priority, and release conditions;

[0115] S303, the NPU loads the information of the network layer to be executed and reads the inter-layer data residency table;

[0116] S304, determine whether the current layer input features have been stored in the on-chip resident cache pool. If yes, execute S305; otherwise, execute S306.

[0117] S305, directly read the input features from the on-chip resident buffer pool, and execute S307;

[0118] S306, the DMA controller is started to load input features from external memory to the input buffer window via the data transfer area;

[0119] S307, Read the current layer weight block and store it in the weight cache window;

[0120] S308, the weight block in the weight buffer window is extracted by the weight broadcast controller and sent into the multicast network;

[0121] S309, the multicast network synchronously distributes weights to all PE computing clusters;

[0122] S310, each PE computation cluster combines input features and broadcast weights to complete convolutional inference computation in parallel;

[0123] S311, After the operation is completed, the output feature map is written to the output buffer window;

[0124] S312, Read the inter-layer data residence table to determine whether the output feature map will be reused by subsequent network layers. If so, execute S314; otherwise, execute S313.

[0125] S313 writes the output feature map directly back to external memory via the DMA controller and then jumps back to read the next layer network information.

[0126] S314 writes the output feature map into the on-chip resident cache pool;

[0127] S315, Register the total number of times the feature map is used and the release conditions with the feature map lifecycle manager;

[0128] S316, Determine whether this read operation is the last use of the resident feature map. If yes, execute S317; otherwise, execute S318.

[0129] S317 triggers the cache release controller to release the on-chip cache space occupied by the feature map, reclaims the storage space, and then jumps to the next network layer to loop and execute the whole process until all network layers of the neural network have finished inference.

[0130] S318 keeps features continuously residing in the cache pool, and the process jumps directly to the next network layer for execution.

[0131] The steps disclosed in this embodiment fully cover the entire process of offline model compilation, data loading, weight broadcast calculation, output residency determination, and cache resource reclamation. By using branch logic to distinguish the data processing methods of reused and non-reusable features, the software scheduling level works with the hardware architecture to reduce repeated read and write operations on external storage, and fully leverages the technical advantages of weight broadcast multi-cluster sharing and inter-layer feature on-chip residency and reuse.

[0132] This embodiment uses a three-layer network as an example to illustrate how to reduce external memory access. The output feature map of the first layer is immediately used by the second layer and is also used again in the residual connection of the third layer. In the traditional scheme, the output of the first layer is written back to external memory, then read in by the second layer, and then read in again during the residual connection of the third layer. With this invention, the output of the first layer resides directly in the on-chip cache, and is not released after being read once by the second layer until the residual connection of the third layer is completed.

[0133] Figure 4 A timing diagram of interlayer feature map dwell time is shown according to an example embodiment.

[0134] like Figure 4As shown, the system is horizontally divided into six independent timing channels: the first network layer, the resident buffer pool, the feature map lifecycle manager, the second network layer, the residual branch / third network layer, and external memory. Vertically, all data interaction actions are displayed sequentially according to time. The first network layer performs inference operations first. After the operation is completed, it writes its output intermediate feature map directly into the resident buffer pool for on-chip storage. At the same time, the first network layer sends a registration instruction to the feature map lifecycle manager, indicating that the resident feature has been used twice, and is provided to the second network layer and the residual branch for reading respectively. Then, the second network layer starts inference calculation and reads the first layer resident feature from the resident buffer pool as input data. After the feature map lifecycle manager detects this read operation, it decrements the remaining usage count of the feature by one. After the second network layer completes its operation, it generates a new output feature and writes it into the resident buffer pool synchronously. The reuse rules are registered, and the timing progresses to the residual branch to perform residual fusion operations. The residual branch retrieves the first-layer output features from the resident cache pool again to participate in the calculation. The feature map lifecycle manager decrements the remaining usage times to zero again, and then issues a cache release command to the cache release controller. The resident cache pool reclaims all the storage space occupied by the first-layer output features. In the entire timing process, only the final output features of the neural network are transferred from the output cache window to external memory through the DMA data transfer path. All intermediate feature maps participating in cross-layer and cross-residual branch reuse only circulate within the on-chip resident cache pool and do not have any read / write interaction with external memory. This allows intermediate features to be retained in the on-chip cache throughout the residual network inference process, eliminating the need for repeated write-back and read-from external memory operations of intermediate features, and minimizing the external memory bandwidth usage and memory access power consumption during residual model inference.

[0135] Example 3:

[0136] Example 3 is a practical application example of Examples 1 and 2 in engineering:

[0137] (1) Interlayer feature map dwelling in continuous convolutional networks

[0138] In an edge-side image recognition model, the network includes consecutive convolutional layers, activation layers, and pointwise convolutional layers. Traditional NPUs write the output feature map back to external memory after the first convolutional layer; during the execution of the second layer, the output is read back to the on-chip input buffer via DMA. With this invention, the model compiler analyzes the network structure offline and discovers that the output of the first convolutional layer will be immediately used by the second layer; therefore, it marks this output as "on-chip resident" in the inter-layer data residency table.

[0139] During inference, the output of the first convolutional layer is written directly to the on-chip resident cache, and the remaining usage count is recorded as one. During the execution of the second layer, the input data is read directly from the resident cache, without needing to be reloaded from external memory. After the second layer read is complete, the lifecycle manager reduces the remaining usage count to zero and releases the cache space occupied by the output of the first convolutional layer. In this way, the intermediate feature map avoids one external write-back and one external read-in.

[0140] (2) Cross-layer reuse in residual networks

[0141] In a neural network with residual connections, the output of the first layer not only serves as the input to the second layer but is also added to the main branch output in the fourth layer using residuals. With this invention, the inter-layer data residency table records that the output of the first layer is used twice, with subsequent uses occurring in the second and fourth layers.

[0142] After the first layer's calculation is completed, the output feature map is written to the resident cache pool. After the second layer reads this feature map, the lifecycle manager reduces the remaining usage count from two to one, but does not release the cache. When the fourth layer performs residual addition, it reads the feature map again. After reading, the remaining usage count becomes zero, and the cache release controller releases the space. This embodiment illustrates that the present invention can support data resident across layers and branches.

[0143] (3) Weight broadcasting of multi-PE computation clusters

[0144] In a convolutional layer, multiple PE computation clusters need to use the same convolutional kernel weight block to compute different output positions. In traditional schemes, each PE computation cluster reads the same weight block from the weight cache, resulting in duplicate reads. With this invention, the weight broadcast controller reads the weight block from the weight cache window only once and writes it to the weight broadcast buffer. Subsequently, the multicast network distributes the weight block to multiple PE computation clusters simultaneously.

[0145] After receiving the same weight block, each PE computing cluster performs multiplication and addition calculations with different input feature blocks. This method reduces the number of weight cache accesses, lowers on-chip data transfer power consumption, and improves the synchronous execution efficiency of multiple computing clusters.

[0146] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An NPU on-chip storage and weight broadcast system based on inter-layer data residency, characterized in that, It includes a main processor / model compiler, on-chip configuration and data bus, and functional areas, which include an inter-layer resident management area, an on-chip storage area, a data transport area, a weight broadcast area, an NPU computing array area, and an output and recycling area; The main processor / model compiler is bidirectionally connected to the functional area via the on-chip configuration and data bus; The inter-layer residency management area is used to record the data dependencies between different network layers; the inter-layer residency management area includes an inter-layer data residency table, a feature map lifecycle manager, and a cache space allocator; The inter-layer residency management area uniformly manages cache allocation, data residency determination, and cache release instruction issuance based on the inter-layer data dependency relationship generated by model compilation. The inter-layer residency management area outputs cache space scheduling configuration to the on-chip storage area. The inter-layer residency management area issues DMA data transfer control strategy to the data transfer area. The inter-layer residency management area provides layer switching timing signals for the weight broadcast area. The on-chip storage area includes an on-chip resident cache pool, an input cache window, a weight cache window, and an output cache window; the on-chip resident cache pool is specifically used to store intermediate feature maps reused across layers, and the space of the on-chip resident cache pool is dynamically managed by the cache space allocator; the input cache window caches the input features of the current layer, the weight cache window caches the convolution weights, and the output cache window caches the temporary output results of the array; The data transfer area is used to move input data and weight data from external memory into the on-chip storage area, and write the final output data or data that cannot be resident back to external memory. The data transfer area includes an external memory interface and a DMA controller. The data transfer area only completes bidirectional data transfer between external memory and the on-chip storage area when the inter-layer resident management area determines that the input data is not resident on the chip or the output data has no reuse value. The weight broadcast area includes a weight broadcast controller, a weight broadcast buffer, and a multicast network. The weight broadcast controller reads weight blocks from the weight cache window. After the weight broadcast controller sends the weight blocks into the weight broadcast buffer to complete alignment and fragmentation, the weight broadcast controller distributes the same weight to multiple parallel PE computing clusters in the NPU computing array area through the multicast network. The NPU computing array area includes several PE computing clusters. The computing clusters read feature map data from the input buffer window or the on-chip resident buffer pool, receive weight data from the multicast network, perform convolution, matrix multiplication or fully connected computation, and write the computation results back to the output buffer window and transmit them to the output and recycling area. The output and recycling area includes an output write-back controller and a cache release controller; the output write-back controller is used to determine the destination of the current layer output data; the cache release controller receives the release instruction issued by the feature map lifecycle manager and reclaims the space occupied by the on-chip resident cache pool after the remaining usage count of the resident feature is cleared to zero.

2. The NPU on-chip storage and weight broadcasting system based on inter-layer data residency according to claim 1, characterized in that, A portion of SRAM is used as a dynamically allocable on-chip resident cache pool, which dynamically adjusts its cache space based on the network layer execution order and data lifecycle.

3. The NPU on-chip storage and weight broadcasting system based on inter-layer data residency according to claim 2, characterized in that, The inter-layer data residency table is used to implement inter-layer data residency control; the inter-layer data residency table is generated offline by the model compiler, or written to the NPU by the main processor when the model is loaded; Each network layer corresponds to one or more entries in the inter-layer data residency table, and each entry includes the following fields: The current layer number is used to identify the network layer that generated the intermediate feature map; Output feature map identifier, which is used to identify the intermediate feature map generated in the current layer; A list of subsequent layers, which records which subsequent layers will use the feature map; Remaining usage count, which records how many more times the feature map needs to be read; Retention priority, which indicates whether the feature map is worth keeping in the on-chip cache; On-chip cache address, which is used to record the storage location of the feature map in the on-chip resident cache pool; The residency size is used to record the cache capacity occupied by the feature map; Release conditions, which indicate that the feature map is released immediately after its last use, or after the specified layer ends; A write-back flag, which indicates whether the feature map needs to be written back to external memory; When executing each layer, the NPU uses the inter-layer data residency table to determine whether the current layer input is already resided in the on-chip cache, whether the current layer output should continue to be retained, and which old feature maps can be released.

4. The NPU on-chip storage and weight broadcasting system based on inter-layer data residency according to claim 1, characterized in that, The feature map lifecycle manager is used to track the entire process of intermediate feature maps from generation, residence, reuse to release. When a certain network layer has completed the calculation, the output write-back controller submits the output feature map information to the lifecycle manager. The lifecycle manager queries the inter-layer data residence table to determine whether the feature map will be used by subsequent layers. If the feature map will continue to be used in the next layer or subsequent short-range layers, the lifecycle manager marks it as a resident state and requests the cache space allocator to allocate resident cache space for it. If the feature map will no longer be used, the lifecycle manager will directly release the feature map or write it back to external storage. If the feature map needs to be reused by residual branches, jump branches, or multi-branch networks, the lifecycle manager records its remaining usage count and decrements the count after each read. When the remaining usage count is zero, the cache is released.

5. The NPU on-chip storage and weight broadcasting system based on inter-layer data residency according to claim 1, characterized in that, For different convolution kernels, different output channels, or different matrix blocks, the weight broadcast controller broadcasts sequentially according to the weight block number; for weight blocks used only by a single computation cluster, the weight broadcast controller degenerates into unicast transmission.

6. The NPU on-chip storage and weight broadcasting system based on inter-layer data residency according to claim 1, characterized in that, The on-chip resident cache pool is controlled by the feature map lifecycle manager; after a single layer ends, the on-chip resident cache pool is not immediately written back or overwritten, but is retained according to the actual dependencies of subsequent network layers; The on-chip resident cache pool uses either a contiguous address allocation method or a block-page allocation method.

7. The NPU on-chip storage and weight broadcasting system based on inter-layer data residency according to claim 1, characterized in that, After each layer of calculation is completed, the output is written back to the controller to determine whether there is a need for subsequent reuse of the array output characteristics based on the inter-layer data residency table. If the current layer output will be used immediately by the next layer, the output write-back controller will write the current layer output directly to the resident buffer pool or the next layer input buffer window, without writing it back to external memory. If the current layer output will be used by a subsequent branch, but not immediately by the next layer, the output write-back controller writes the current layer output into the resident cache pool and sets the corresponding remaining number of uses. If the current layer output will not be used by subsequent layers, but is the final output, the output write-back controller writes the current layer output into the output buffer window and writes the current layer output data back to the external memory through the DMA controller. If the current layer output size is too large and the on-chip cache cannot be fully resident, the output write-back controller adopts a block resident strategy, retaining only the tiles that need to be used in the short term, and writing the rest back to external memory.

8. The NPU on-chip storage and weight broadcasting system based on inter-layer data residency according to claim 1, characterized in that, The multicast network is used to distribute weight blocks in the weight broadcast buffer to multiple PE computing clusters. The multicast network adopts a bus broadcast method, a tree broadcast method, a cross-switch broadcast method, or an on-chip network multicast method. The bus broadcast method involves the weight broadcast controller placing weight blocks on a shared broadcast bus, which are then received by each PE computing cluster according to an enable signal. The tree broadcast method involves the weight data being first sent to several intermediate nodes, which then distribute it to multiple computing clusters. The cross-switch broadcast method or the on-chip network multicast method involves the weight broadcast controller sending weight blocks to a designated computing cluster according to the target computing cluster mask.

9. An NPU on-chip storage and weight broadcasting method based on inter-layer data residency, used in the NPU on-chip storage and weight broadcasting system of any one of claims 1-8, characterized in that, Includes the following steps: The model compiler or main processor analyzes the inter-layer dependencies of the neural network model and generates an inter-layer data residency table; The NPU loads the current layer's input data and weight data; it determines whether the current layer's input features are already stored in the on-chip resident cache pool. If so, it reads the input features directly from the on-chip resident cache pool; otherwise, it starts the DMA controller to load the input features from external memory to the input cache window via the data transfer area. After the input data preparation is complete, the current layer weight block is read and stored in the weight cache window; The weight broadcast controller extracts the weight blocks in the weight cache window and sends them to the multicast network. The multicast network then synchronously distributes the weights to all PE computing clusters. Each PE computation cluster combines input features and broadcast weights to complete convolutional inference computation in parallel; After the operation is completed, the output characteristics are written to the output buffer window; After the current layer completes the calculation, the output write-back controller determines whether the output feature needs to be used by subsequent layers. If it needs to be used by subsequent layers, it is written into the on-chip resident cache pool, and the lifecycle manager registers the number of times the output feature is used. If it does not need to be used by subsequent layers, the output feature is directly written back to external memory through the DMA controller or the cache release controller releases the on-chip cache space occupied by the output feature. When a subsequent layer reads an output feature residing in the on-chip resident cache pool, the lifecycle manager decrements the remaining usage count of the output feature by one; if the remaining usage count is zero, the cache release controller releases the on-chip cache space occupied by the output feature.

Images