Hierarchical storage system for large model inference acceleration chip

Abstract

The application discloses a hierarchical storage system for a large model inference acceleration chip, which comprises a hierarchical storage module, a data label generation unit, a data hierarchical control unit, a hotspot expert identification unit, a data recycling unit and a monitoring and protection unit. The hierarchical storage module caches the data block sent by the AFD cooperative interface module in the pre-storage partition, the data label generation unit generates the data label of the target data block, and updates the record expert calculation unit and the storage address when a new data block is detected. The data hierarchical control unit copies the target data block to the matching data temperature partition according to the data label. After the hotspot expert identification unit identifies the hotspot expert calculation unit, the hotspot expert parameter copy is generated and saved to the hotspot copy partition. The data recycling unit is used for identifying and recycling aged data. The monitoring and protection unit is used for identifying the over-limit partition and triggering emergency recycling. The scheme realizes storage scheduling through a full hardware circuit, eliminates bandwidth contention and access bottleneck, and supports ultra-low delay inference.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit technology, and in particular to a hierarchical storage system for large-model inference acceleration chips. Background Technology

[0002] As the scale of large language model parameters approaches the trillion level, the Mixture-of-Experts (MoE) architecture supports the continuous expansion of model capacity while controlling computational costs through sparse activation mechanisms. However, the surge in the number of experts makes on-chip storage access efficiency and bandwidth utilization the core bottlenecks for inference latency. The autoregressive decoding stage accounts for a high proportion of the total inference time, and the frequent reads and writes of the key-value cache (KV Cache) and the dynamic loading of expert weights place extremely high demands on storage real-time performance. The Attention-FFN Disaggregation (AFD) architecture decomposes the decoding part into three modular units: attention, feed-forward network (FFN), and MoE. These units are executed independently on a dedicated Neural Processing Unit (NPU) to achieve hardware decoupling. However, the data prefetching, residence, and recycling during the independent operation of each unit urgently need to achieve precise timing matching with the AFD asynchronous inference pipeline, which poses a new challenge to the hardwareization and intelligence level of the storage scheduling circuit.

[0003] Existing technologies primarily rely on software-driven storage scheduling schemes. At the storage management level, allocation and reclamation are achieved through operating system page table management, cache coherency protocols, and user-space key-value cache managers. For MoE expert management, Expert Parallelism (EP) is employed to distribute experts across multiple graphics processors. CPU memory offloading and on-demand loading address insufficient video memory. Some schemes use software prefetching or dynamic load balancing algorithms to identify hot experts and allocate redundant copies at runtime. Key-value cache management introduces eviction and recomputation strategies based on attention mechanisms. Overall, existing solutions lack dedicated hardware storage scheduling circuitry for AFD-MoE inference chips.

[0004] Existing technologies have significant drawbacks: First, scheduling relies on software page tables, consistency protocols, and memory managers, lacking dedicated hardware circuitry. This results in microsecond-level scheduling latency and additional overhead, making it difficult to meet the timing requirements of ultra-low latency inference. Second, the hotspot expert effect in the MoE model is only addressed at the software level, lacking hardware-level hotspot identification and replica scheduling circuitry. This prevents real-time load balancing, easily creating bandwidth bottlenecks and reducing computing power utilization. Third, historical features accumulate continuously, and the lack of dedicated hardware aging and recycling circuitry prevents the automatic removal of expired data, leading to on-chip memory overflow and forcing frequent off-chip data swapping, further increasing latency. In summary, there is an urgent need for a purely hardware-based, high-timing-efficiency, and AFD-MoE-compatible intelligent hierarchical storage hot and cold scheduling circuit device. Summary of the Invention

[0005] This invention provides a hierarchical storage system for large model inference acceleration chips to solve the problems of high latency, data contention at different call frequencies, long context data overflow, and timing mismatch caused by storage scheduling dependence on software in AFD-MoE inference chips.

[0006] According to one aspect of the present invention, a hierarchical storage system for large model inference acceleration chips is provided, comprising: a hierarchical storage module, a data tag generation unit, a data hierarchical management and control unit, a hotspot expert identification unit, a data recycling unit, and a monitoring and protection unit, wherein the hierarchical storage module includes a first-level cache and a second-level cache; The hierarchical storage module is connected to the attention and feedforward network separation architecture AFD collaborative interface module outside the system, and is used to cache the data blocks sent in real time by the AFD collaborative interface module in the pre-storage partition; the hotspot expert identification unit is connected to the computing module located outside the system, and the computing module includes a hybrid expert routing unit and multiple expert computing units. The data tag generation unit is used to generate a data tag matching the target data block and send it to the data hierarchical control unit when the target data block in the pre-storage partition meets the data copy conditions; and when a new data block is detected in the pre-storage partition, to record the expert computing unit that uses the new data block and the storage address of the new data block in the pre-storage partition in the expert address table. The data classification and control unit is used to copy the target data block to the matching data temperature partition for compressed storage according to the tag type of the received data tag, and to provide the partition mapping information of the target data block from the pre-storage partition to the matching data temperature partition to the hot spot expert identification unit and the data recycling unit; wherein, the data temperature partition includes cold data partition, hot data partition and warm data partition; The hotspot expert identification unit is used to identify the hotspot expert computing units in the computing module based on the expert routing information table sent in real time by the hybrid expert routing unit, and to generate hotspot expert parameter copies by obtaining each data block belonging to the hotspot expert computing unit from each data temperature partition according to the expert address table and the mapping information of each partition, and saving them to the hotspot copy partition; among them, the hot data partition, the temperature data partition and the hotspot copy partition are recyclable partitions. The data recycling unit is used to identify aging data in each recyclable partition based on the write timestamp of each recyclable data block in each recyclable partition and the aging data identification conditions of each recyclable partition, and to recycle the aging data in the background according to the mapping information of each partition, so as to release the space of each recyclable partition. The monitoring and protection unit is used to identify over-limit partitions based on the real-time space occupancy rate of each recyclable partition, adjust the aging data identification conditions and / or compression storage parameters of the over-limit partitions, and trigger emergency recycling for the over-limit partitions.

[0007] The technical solution of this invention is a hierarchical storage system for large-model inference acceleration chips, including a hierarchical storage module, a data tag generation unit, a data hierarchical management and control unit, a hotspot expert identification unit, a data recycling unit, and a monitoring and protection unit. The hierarchical storage module includes a level 1 cache and a level 2 cache. The hierarchical storage module is connected to an external AFD collaboration interface module and is used to cache data blocks sent in real time by the AFD collaboration interface module in a pre-storage partition. The hotspot expert identification unit is connected to a computing module located outside the system. The computing module includes a hybrid expert routing unit and multiple expert computing units. The data tag generation unit generates matching data tags and sends them to the data hierarchical management and control unit when a target data block in the pre-storage partition meets the data copy conditions. When a new data block is detected in the pre-storage partition, the expert computing unit using the new data block and the storage address of the new data block in the pre-storage partition are recorded in the expert address table. The data classification and control unit is used to copy target data blocks to matching data temperature partitions for compressed storage based on the tag type of the received data tags. It also provides the partition mapping information from the pre-storage partition to the matching data temperature partition to the hotspot expert identification unit and the data recycling unit. The data temperature partitions include cold data partitions, hot data partitions, and warm data partitions. The hotspot expert identification unit identifies hotspot expert calculation units based on the expert routing information table sent in real-time by the hybrid expert routing unit. It then retrieves data blocks belonging to the hotspot expert calculation units from each data temperature partition based on the expert address table and partition mapping information, generating hotspot expert parameter copies and saving them to the hotspot copy partition. The data recycling unit identifies aging data based on the write timestamps of recyclable data blocks in each recyclable partition and the aging data identification conditions of each recyclable partition. It then performs background recycling of the aging data based on the partition mapping information to free up space in each recyclable partition. The monitoring and protection unit identifies over-limit partitions based on the real-time space occupancy rate of each recyclable partition, adjusts the aging data identification conditions and / or compression storage parameters of the over-limit partitions, and triggers emergency reclamation for the over-limit partitions. Hot data partitions, warm data partitions, and hot replica partitions are recyclable partitions. This technical solution implements storage scheduling entirely through hardware circuitry, avoiding the overhead of software page table management and cache consistency protocols. Through physical partitioning and isolation of cold, warm, and hot data, and dynamic expert replica distribution for hot data, bandwidth contention and access bottlenecks are eliminated. Combined with a hardware-level sliding aging reclamation mechanism, storage overflow is prevented. Simultaneously, time-series collaborative control and deep matching with the AFD inference pipeline achieve efficient synchronization of data prefetching and computation, effectively supporting ultra-low latency large model inference scenarios.

[0008] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of the structure of a hierarchical storage system for a large-model inference acceleration chip according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of a hierarchical storage system for a large-model inference acceleration chip provided according to Embodiment 2 of the present invention. Detailed Implementation

[0011] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0012] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0013] Example 1 Figure 1 This is a schematic diagram of a hierarchical storage system 100 for large-model inference acceleration chips provided in Embodiment 1 of the present invention. This embodiment is applicable to on-chip inference acceleration in large-model inference chips, which implements nanosecond-level storage scheduling, physical partitioning of hot and cold data, dynamic distribution of hot expert data, and automatic aging and reclamation of long contexts using all-hardware circuitry. Figure 1As shown, the system includes: a hierarchical storage module 110, a data tag generation unit 120, a data hierarchical management and control unit 130, a hotspot expert identification unit 140, a data recycling unit 150, and a monitoring and protection unit 160. The hierarchical storage module 110 includes a first-level cache 1101 and a second-level cache 1102.

[0014] The hierarchical storage module 110 is connected to the attention and feedforward network separation architecture AFD collaboration interface module 170 outside the system, and is used to cache the data blocks sent in real time by the AFD collaboration interface module 170 in the pre-storage partition. The hotspot expert identification unit 140 is connected to the computing module 180 located outside the system. The computing module 180 includes a hybrid expert routing unit and multiple expert computing units.

[0015] The AFD collaboration interface module 170 can be an interface module connecting the attention computing unit and the feedforward network computing unit in the AFD architecture. It receives input data from the AFD inference pipeline and sends data blocks to be stored to the hierarchical storage module 110 in real time. The pre-storage partition refers to a physical storage area partitioned within the hierarchical storage module 110, used for temporarily caching data blocks sent by the AFD collaboration interface module 170. The hybrid expert routing unit refers to a hardware routing circuit located in the computing module 180. It receives input tokens, calculates the activation weights of each hybrid expert (MoE expert) based on a preset gating network, and outputs a route hit signal to the hierarchical storage module 110 and the hotspot expert identification unit 140. The expert computing unit refers to a dedicated computing circuit located in the computing module 180 for performing feedforward network operations on a single hybrid expert (MoE expert). The computing module refers to a set of computing cores located outside the hierarchical storage system, including the hybrid expert routing unit and multiple expert computing units, used for expert routing decisions and parallel computation by the expert computing units during the inference process of the large model inference acceleration chip.

[0016] Understandably, the hierarchical storage module 110 is connected to the AFD collaboration interface module 170, receiving data blocks sent by the AFD collaboration interface module 170 via the data bus. These data blocks may contain activation value data output by the attention calculation unit in the AFD inference pipeline, along with corresponding token identification information. After receiving the data blocks, the hierarchical storage module 110 can cache them in the corresponding pre-storage partition. Additionally, the hotspot expert identification unit 140 is connected to the computing module 180 located outside the system. The computing module 180 includes a hybrid expert routing unit and multiple expert computing units. The hotspot expert identification unit 140 can receive the routing hit signals of each hybrid expert output by the hybrid expert routing unit in real time. The multiple expert computing units within the computing module 180 execute expert network calculations in parallel under the scheduling of the hybrid expert routing unit.

[0017] The data tag generation unit 120 is used to generate a data tag matching the target data block and send it to the data hierarchical control unit 130 when the target data block in the pre-storage partition meets the data copy conditions, and to record the expert computing unit that uses the new data block and the storage address of the new data block in the pre-storage partition in the expert address table when a new data block is detected in the pre-storage partition.

[0018] The target data block can refer to the data block to be processed that is stored in the pre-storage partition, whose data tag is determined after data classification operations, and which meets the data copy triggering conditions. The expert address table can refer to the data lookup table set inside the data tag generation unit 120, which can be used to record the mapping relationship between the expert calculation unit corresponding to each hybrid expert and the storage address of the required data block in the pre-storage partition.

[0019] Understandably, the data tag generation unit 120 continuously monitors the residency status of each data block in the pre-storage partition. When a data block is detected to meet data copy conditions, such as when the access frequency of the data block in the pre-storage partition reaches a preset threshold, the data block can be identified as the target data block. The data tag generation unit 120 can automatically generate a data tag matching the target data block according to preset rules and send the data tag to the data hierarchical management unit 130. In addition, the data tag generation unit 120 continuously monitors the write status of the pre-storage partition. When it detects that the AFD collaborative interface module 170 writes a new data block to the pre-storage partition, the data tag generation unit 120 obtains the identification information of the expert computing unit corresponding to the new data block and the storage address of the new data block in the pre-storage partition, and records the above information in the expert address table. To achieve the above functions, the data tag generation unit 120 can integrate a data caching circuit. This circuit can be implemented using pure hardware logic without software intervention. The data caching circuit identifies new data block writing behavior by monitoring the write enable signal and valid address signal of the pre-storage partition in real time. Simultaneously, it captures the expert computing unit identifier transmitted along with the data stream of the AFD inference pipeline, thereby achieving the binding of data blocks with corresponding experts at the hardware level. The data caching circuit constructs an expert address table through a built-in hardware register array. When a new data block is written, it automatically records the corresponding expert identifier and the physical storage address in the pre-storage partition. For scenarios such as address overwriting or data reclamation, the data caching circuit can automatically update or clear the corresponding table entries to ensure that the address mapping relationship is valid in real time. This expert address table can maintain the mapping relationship between expert computing unit identifiers and data block storage addresses in real time. Subsequently, by querying this expert address table, the specific storage location of the data blocks required by each expert computing unit in the pre-storage partition can be quickly located, realizing fast addressing and scheduling of data blocks.

[0020] Optionally, the types of data tags include: cold data tags that match cold data partitions, warm data tags that match warm data partitions, and hot data tags that match hot data partitions; The pre-storage partition and cold data partition are set in the second-level cache 1102, while the hot data partition, hot copy partition and warm data partition are set in the first-level cache 1101 or the second-level cache 1102.

[0021] In this context, "cold data" refers to data blocks with low-frequency access, long lifespan, and low timeliness requirements. In AFD inference scenarios, it can include static weight parameters and basic model parameters of various hybrid experts. "Warm data" refers to data blocks with medium-frequency access, medium lifespan, and timeliness decay characteristics. In AFD inference scenarios, it can include historical key-value caches in long-context inference and modality-encoded feature data in multimodal inference. "Hot data" refers to data blocks with high-frequency access, short lifespan, and extreme sensitivity to access latency. In AFD inference scenarios, it can include real-time activation values ​​in the current inference step and high-frequency hit hybrid expert weight data. Level 1 cache 1101 refers to the storage resource with the lowest access latency in the tiered storage system, and its physical implementation can be static random access memory (SRAM). Level 2 cache 1102 refers to a storage resource in the tiered storage system with a capacity larger than Level 1 cache 1101 and a higher access latency than Level 1 cache 1101, and its physical implementation can be dynamic random access memory (DRAM). A cold data partition refers to a physical storage area in a tiered storage system, defined by hierarchical address mapping, used to store cold data. The data within this partition is protected by address locking and is unaffected by replacement and reclamation operations. A warm data partition refers to a physical storage area in a tiered storage system, defined by hierarchical address mapping, used to store warm data. A hot data partition refers to a physical storage area in a tiered storage system, defined by hierarchical address mapping, used to store hot data with hybrid expert weight parameter replicas.

[0022] It is understood that the data tag types generated by the data tag generation unit 120 correspond one-to-one with the physical storage partition types in the hierarchical storage module 110. Specifically, when the target data block is determined to be cold data, the data tag generation unit 120 generates a cold data tag matching the cold data partition; when determined to be warm data, it generates a warm data tag matching the warm data partition; and when determined to be hot data, it generates a hot data tag matching the hot data partition. Based on the aforementioned data tags, the hierarchical storage module 110 copies the target data block from the pre-storage partition to the corresponding cold data partition, warm data partition, or hot data partition. In terms of storage hierarchy layout, the pre-storage partition serves as the initial write temporary space for data blocks, and its physical location can be set in the secondary cache 1102. The cold data partition serves as a long-term resident area for low-frequency access data, and its physical location is also set in the secondary cache 1102. The hot data partition, hot copy partition, and warm data partition can be physically located in the primary cache 1101 or the secondary cache 1102, depending on the scheduling characteristics such as data access frequency.

[0023] The data classification and control unit 130 is used to copy the target data block to the matching data temperature partition for compressed storage according to the tag type of the received data tag, and to provide the partition mapping information of the target data block from the pre-storage partition to the matching data temperature partition to the hot spot expert identification unit 140 and the data recycling unit 150; wherein, the data temperature partition includes cold data partition, hot data partition and warm data partition.

[0024] Partition mapping information can refer to data information that records the correspondence between the source address and the destination address during the migration of the target data block from the pre-storage partition to the data temperature partition.

[0025] Understandably, after receiving a data tag sent by the data tag generation unit 120, the data classification and control unit 130 can perform data classification and control operations based on the tag type. Specifically, the data classification and control unit 130 can first identify the specific type of the received data tag, i.e., cold data tag, warm data tag, or hot data tag. Then, based on the tag type, it determines the data temperature partition corresponding to the target data block, i.e., cold data tag corresponds to cold data partition, warm data tag corresponds to warm data partition, and hot data tag corresponds to hot data partition. Subsequently, the data classification and control unit 130 copies the target data block from the pre-storage partition to the determined matching data temperature partition, and performs a compressed storage operation on the target data block during the copying process. After the target data block is copied, the data classification and control unit 130 generates partition mapping information, which records the address mapping relationship of the target data block from the pre-storage partition to the matching data temperature partition. The data classification and control unit 130 provides this partition mapping information to the hotspot expert identification unit 140 and the data recycling unit 150, respectively, so that they can locate the specific storage location of the target data block in the data temperature partition based on the mapping information in subsequent operations.

[0026] The hotspot expert identification unit 140 is used to identify the hotspot expert calculation unit in the calculation module 180 according to the expert routing information table sent in real time by the hybrid expert routing unit, and to obtain the data blocks belonging to the hotspot expert calculation unit from each data temperature partition according to the expert address table and the mapping information of each partition to generate hotspot expert parameter copies and save them to the hotspot copy partition; wherein, the hot data partition, the temperature data partition and the hotspot copy partition are recyclable partitions.

[0027] The expert routing information table refers to the data information sent in real time by the hybrid expert routing unit to the hotspot expert identification unit 140, which can be used to record the route hit frequency of each expert calculation unit in the current inference step. The hotspot expert calculation unit refers to an expert calculation unit whose route hit frequency exceeds a preset hardware threshold according to the expert routing information table. The hotspot expert parameter copy refers to the complete set of weight parameters generated by the hotspot expert identification unit 140 after obtaining data blocks belonging to the same hotspot expert calculation unit from each data temperature partition.

[0028] Understandably, the hotspot expert identification unit 140 receives the expert routing information table sent by the hybrid expert routing unit in real time. Based on the routing hit frequency of each expert in the current inference step recorded in the information table, and combined with a preset hardware frequency threshold, it identifies expert computing units whose routing hit frequency exceeds the threshold and marks them as hotspot expert computing units. After identifying the hotspot expert computing unit, the hotspot expert identification unit 140 obtains the storage addresses of each data block required by the hotspot expert computing unit in the pre-storage partition according to the expert address table. At the same time, based on the partition mapping information from the pre-storage partition to each data temperature partition provided by the data hierarchical management unit 130, it locates the specific storage location of each data block in the cold data partition, hot data partition, and warm data partition. The hotspot expert identification unit 140 obtains each data block belonging to the hotspot expert computing unit from each of the above-mentioned data temperature partitions, merges them to generate a complete hotspot expert parameter copy, and saves the parameter copy to the hotspot copy partition. Among them, the hot data partition, warm data partition, and hotspot replica partition are all recyclable partitions. According to the pre-set recycling conditions for each data block in the data partition, the data recycling unit 150 can recycle the data when the recycling conditions are met. The data blocks in the warm data partition are managed by the sliding aging recycling circuit, which automatically eliminates expired data according to the time window. The hotspot expert parameter replicas in the hotspot replica partition are automatically recycled and released by the hotspot expert identification unit 140 after the routing hit frequency of the corresponding expert calculation unit falls below a preset threshold. The cold data partition is not among the recyclable partitions. The data in it is locked and protected, and does not participate in any recycling operations, ensuring that the static weight parameters remain stable for a long time.

[0029] The data recycling unit 150 is used to identify aging data in each recyclable partition based on the write timestamp of each recyclable data block in each recyclable partition and the aging data identification conditions of each recyclable partition, and to recycle the aging data in the background according to the mapping information of each partition, so as to release the space of each recyclable partition.

[0030] Understandably, the data recycling unit 150 receives partition mapping information from the data grading and control unit 130, showing the data blocks' positions from the pre-storage partition to the data temperature partitions, and obtains the write timestamps of each data block stored in each recyclable partition. These write timestamps are recorded by the grading storage module 110 and synchronously provided to the data recycling unit 150 when the data blocks are written to each recyclable partition, serving as a timing benchmark for determining the aging status of the data blocks. The data recycling unit 150 identifies aging data that meets the aging data identification conditions in each recyclable partition. After identifying the aging data, the data recycling unit 150 locates the specific storage location of the aging data in the corresponding recyclable partition based on the partition mapping information provided by the data grading and control unit 130, and performs a background recycling operation. This background recycling is automatically completed in hardware during the idle clock cycles of the inference computing pipeline, clearing aging data and releasing the corresponding storage space. The recycling process does not occupy inference bandwidth or interrupt business data flow.

[0031] The monitoring and protection unit 160 is used to identify over-limit partitions based on the real-time space occupancy rate of each recyclable partition, adjust the aging data identification conditions and / or compression storage parameters of the over-limit partitions, and trigger emergency recycling for the over-limit partitions.

[0032] Here, an over-limit partition refers to a reclaimable partition whose real-time space occupancy rate exceeds a preset occupancy rate threshold. Compression storage parameters refer to the control parameters used by the hierarchical storage module 110 when performing compression storage on the target data block, and may include compression ratio configuration and compression algorithm selection parameters.

[0033] Understandably, the monitoring and protection unit 160 can acquire the real-time space occupancy rate of each recyclable partition and compare it with the corresponding partition's preset occupancy rate threshold to identify over-limit partitions whose real-time space occupancy rate exceeds the preset threshold. After identifying an over-limit partition, the monitoring and protection unit 160 can dynamically tighten the aging data identification conditions for that partition based on its type and degree of over-limitation. Specifically, for warm data partitions, the sliding aging window length can be shortened, allowing the data recycling unit 150 to identify resident data as aging data within a shorter time window. For hot data partitions, a shorter hot data lifecycle threshold can be set to accelerate the recycling triggering of hot data at the end of its lifecycle. For hot replica partitions, the sensitivity of hot replica recycling can be increased, allowing hot expert parameter replicas with decreasing frequency to be identified as aging data more quickly. In addition, based on the degree of over-limitation of the over-limit partition, the monitoring and protection unit 160 can output a compression ratio control signal to the hierarchical storage module 110 to increase the compression ratio when writing subsequent data to that partition, thereby reducing the space occupancy per unit data block and slowing down the space exhaustion rate. At the same time, the monitoring and protection unit 160 can send an emergency recovery trigger signal to the data recovery unit 150, instructing the data recovery unit 150 to prioritize the recovery operation of the over-limit zone.

[0034] The technical solution of this invention is a hierarchical storage system for large-model inference acceleration chips, including a hierarchical storage module, a data tag generation unit, a data hierarchical management and control unit, a hotspot expert identification unit, a data recycling unit, and a monitoring and protection unit. The hierarchical storage module includes a level 1 cache and a level 2 cache. The hierarchical storage module is connected to an external AFD collaboration interface module and is used to cache data blocks sent in real time by the AFD collaboration interface module in a pre-storage partition. The hotspot expert identification unit is connected to a computing module located outside the system. The computing module includes a hybrid expert routing unit and multiple expert computing units. The data tag generation unit generates matching data tags and sends them to the data hierarchical management and control unit when a target data block in the pre-storage partition meets the data copy conditions. When a new data block is detected in the pre-storage partition, the expert computing unit using the new data block and the storage address of the new data block in the pre-storage partition are recorded in the expert address table. The data classification and control unit is used to copy target data blocks to matching data temperature partitions for compressed storage based on the tag type of the received data tags. It also provides the partition mapping information from the pre-storage partition to the matching data temperature partition to the hotspot expert identification unit and the data recycling unit. The data temperature partitions include cold data partitions, hot data partitions, and warm data partitions. The hotspot expert identification unit identifies hotspot expert calculation units based on the expert routing information table sent in real-time by the hybrid expert routing unit. It then retrieves data blocks belonging to the hotspot expert calculation units from each data temperature partition based on the expert address table and partition mapping information, generating hotspot expert parameter copies and saving them to the hotspot copy partition. The data recycling unit identifies aging data based on the write timestamps of recyclable data blocks in each recyclable partition and the aging data identification conditions of each recyclable partition. It then performs background recycling of the aging data based on the partition mapping information to free up space in each recyclable partition. The monitoring and protection unit identifies over-limit partitions based on the real-time space occupancy rate of each recyclable partition, adjusts the aging data identification conditions and / or compression storage parameters of the over-limit partitions, and triggers emergency reclamation for the over-limit partitions. Hot data partitions, warm data partitions, and hot replica partitions are recyclable partitions. This technical solution implements storage scheduling entirely through hardware circuitry, avoiding the overhead of software page table management and cache consistency protocols. Through physical partitioning and isolation of cold, warm, and hot data, and dynamic expert replica distribution for hot data, bandwidth contention and access bottlenecks are eliminated. Combined with a hardware-level sliding aging reclamation mechanism, storage overflow is prevented. Simultaneously, time-series collaborative control and deep matching with the AFD inference pipeline achieve efficient synchronization of data prefetching and computation, effectively supporting ultra-low latency large model inference scenarios.

[0035] Example 2 Figure 2This is a schematic diagram of a hierarchical storage system 200 for a large-model inference acceleration chip provided in Embodiment 2 of the present invention. This embodiment is a specific embodiment based on the above embodiment, including: the specific functions of each sub-circuit of the hierarchical storage system constituent units. For example... Figure 2 As shown, it includes: a data tag generation unit 210, a data hierarchical management and control unit 220, a hotspot expert identification unit 230, a data recycling unit 240, a monitoring and protection unit 250, a time-series collaborative control unit 260, a hierarchical storage module 290, and a hardware compression and decompression engine 2010.

[0036] The data tag generation unit 210 includes: an access frequency statistics circuit 2101, a four-dimensional feature acquisition circuit 2102, a tag generation logic circuit 2103, and a threshold configuration circuit 2104; The access frequency statistics circuit 2101 is used to start timing after any data block is stored in the pre-storage partition, and when the timing reaches a preset fixed time window, determine that the data block is the target data block that meets the data copying conditions, and output the access frequency signal according to the number of accesses to the target data block counted by the hardware timer in the fixed time window. The four-dimensional feature acquisition circuit 2102 is used to simultaneously acquire the service attribute signal, life cycle signal and pipeline timing signal of the target data block when the access frequency signal of the target data block is received, and form a four-dimensional feature signal with the access frequency signal, and send the four-dimensional feature signal to the tag generation logic circuit 2103. Tag generation logic circuit 2103 is used to generate data tags that match the target data block based on the four-dimensional feature signal of the target data block and at least one pre-set access frequency threshold. Threshold configuration circuit 2104 is used to configure at least one access frequency threshold that matches a fixed time window, according to the needs of the computing task.

[0037] The preset fixed time window can refer to a fixed-duration statistical period set by a hardware timer in the access frequency statistics circuit 2101. The length of this time window is configured through a hardware register and can be used to limit the time range for statistically analyzing the number of accesses to the data block. The service attribute signal can refer to a signal sent by the hybrid expert routing unit to the four-dimensional feature acquisition circuit 2102 via hardwired connection, which can be used to indicate the data type of the target data block and its functional role in inference computation. The lifecycle signal can refer to the timeliness requirements of the target data block in the inference process and the expected reusable window length. The pipeline timing signal can refer to the stage position and timing alignment information of the target data block in the AFD inference pipeline. The access frequency threshold can refer to a preset hardware-configurable threshold in the threshold configuration circuit 2104, which can be used to compare with the statistical access count of the target data block to determine its hot or cold attributes. This threshold can include a high-frequency threshold, a mid-frequency threshold, and a low-frequency threshold, corresponding to the judgment boundaries of hot data, warm data, and cold data, respectively.

[0038] Specifically, the data tag generation unit 210 may include an access frequency statistics circuit 2101, a four-dimensional feature acquisition circuit 2102, a tag generation logic circuit 2103, and a threshold configuration circuit 2104. The access frequency statistics circuit 2101 starts a hardware timer after any data block is stored in the pre-storage partition. When the timer reaches a preset fixed time window, the data block is identified as the target data block, triggering the access frequency statistics operation. The access frequency statistics circuit 2101 outputs an access frequency signal based on the number of accesses to the target data block counted by the hardware timer within the fixed time window. After receiving the access frequency signal output by the access frequency statistics circuit 2101, the four-dimensional feature acquisition circuit 2102 can acquire the service attribute signal, lifecycle signal, and pipeline timing signal of the target data block via hardwired connections. These three signals, along with the received access frequency signal, form a four-dimensional feature signal, which is then sent to the tag generation logic circuit 2103. The tag generation logic circuit 2103 can receive the four-dimensional feature signal sent by the four-dimensional feature acquisition circuit 2102. Based on four types of information—access frequency signal, business attribute signal, lifecycle signal, and pipeline timing signal—of the target data block, and combined with at least one access frequency threshold preset in the threshold configuration circuit 2104, it performs hardware-level logic operations through combinational logic gates to automatically generate data tags matching the target data block. These tags can include cold data tags, warm data tags, or hot data tags, and can be sent to the data hierarchical management unit 220. The threshold configuration circuit 2104 can be used to configure hardware registers according to the needs of the computational task, setting at least one access frequency threshold matching a preset fixed time window. This can include high-frequency thresholds, mid-frequency thresholds, and low-frequency thresholds, and can be used to adapt to the hotspot feature determination requirements of hybrid expert models of different scales.

[0039] The data hierarchical management and control unit 220 includes: a hardware lookup table address mapping circuit 2201, a partition isolation gating circuit 2202, a cold data latch circuit 2203, and an address verification circuit 2204; The hardware lookup table address mapping circuit 2201 is used to obtain the matching data temperature partition according to the tag type of the received data tag, and determine the compressed storage address of the target data block in the matching data temperature partition according to the original storage address of the target data block in the pre-storage partition; after establishing the partition mapping information between the original storage address and the compressed storage address, the target data block is copied to the compressed storage address; The partition isolation gating circuit 2202 is used to set an independent hardware gating gate for each data temperature partition, allowing access to the storage address of the corresponding data temperature partition only based on the data tag; The cold data latch circuit 2203 is used to latch the address range of the cold data partition after the hierarchical storage module is powered on and initialized, to block the replacement and reclamation instructions for the cold data partition, and to perform corresponding address range latching operations on newly added cold data. The address verification circuit 2204 is used to perform real-time verification of the original storage address of the target data block stored in the compressed storage address based on the constructed partition mapping information.

[0040] Here, a hardware gating gate refers to an independent hardware control switch circuit set up for each data temperature partition. Only data blocks carrying the corresponding tag type can trigger the gating signal of that partition, realizing the matching access between the data block and the storage partition. A replacement instruction refers to a control instruction used to replace existing data in a storage address with new data. A reclamation instruction refers to a control instruction used to release the storage space occupied by data in a storage address that meets preset release conditions.

[0041] Specifically, the data hierarchical management unit 220 may include a hardware lookup table address mapping circuit 2201, a partition isolation gating circuit 2202, a cold data latching circuit 2203, and an address verification circuit 2204. The hardware lookup table address mapping circuit 2201, upon receiving a data tag from the data tag generation unit 210, can obtain the matching data temperature partition based on the tag type. Simultaneously, it can calculate and determine the compressed storage address of the target data block in the matching data temperature partition based on the original storage address of the target data block in the pre-storage partition and the attribute information of the target data block. After establishing the partition mapping information between the original storage address and the compressed storage address, the hardware lookup table address mapping circuit 2201 copies the target data block from the pre-storage partition to the location indicated by the compressed storage address. The partition isolation gating circuit 2202 can be used to set independent hardware gating gates for cold data partitions, hot data partitions, and warm data partitions. This hardware gating gate can control access based on the data tag type of the target data block, allowing only target data blocks matching the current data tag to access the storage address of the corresponding data temperature partition. Target data blocks whose tag type does not match the partition type cannot trigger the gating signal, thus physically preventing cross-partition access and avoiding bandwidth contention between data blocks with different temperature attributes. The cold data latch circuit 2203 can automatically hardware latch the address range of the cold data partition after the hierarchical storage module is powered on and initialized, shielding it from any externally input replacement and reclamation commands targeting the cold data partition, ensuring that the data within the cold data partition is not affected by space management operations. Simultaneously, for newly added cold data after initialization, the cold data latch circuit 2203 can synchronously perform the corresponding address range latching operation when the new cold data is written to the cold data partition, ensuring that all cold data addresses are locked and protected. The address verification circuit 2204 can be used to perform real-time hardware verification on the original storage address of the target data block stored in the compressed storage address after the target data block is stored in the compressed storage address, based on the partition mapping information constructed by the hardware lookup table address mapping circuit 2201, to confirm that the target data block has been correctly written to the matching data temperature partition. The verification delay can be controlled within the nanosecond level to avoid data access abnormalities caused by address errors.

[0042] The hotspot expert identification unit 230 includes: a hotspot identification counter 2301, a hardware direct memory access transfer circuit 2302, a replica scheduling circuit 2303, and a replica recycling circuit 2304; Hotspot identification counter 2301 is used to obtain the number of route hits of each expert computing unit according to the expert routing information table sent in real time by the hybrid expert routing unit. When it is determined that the number of route hits of any expert computing unit exceeds the preset hotspot expert threshold, the expert computing unit is identified as a hotspot expert computing unit, and the expert identifier of the expert computing unit is transmitted to the hardware direct memory access transfer circuit 2302. The hardware direct memory access transfer circuit 2302 is used to query the expert address table and the mapping information of each partition according to the received expert identifier, and to obtain each data block belonging to the hot spot expert calculation unit from each data temperature partition according to the query result to generate a hot spot expert parameter copy and save it to the hot spot copy partition. The replica scheduling circuit 2303 is used to dynamically schedule the access requests for hot expert parameters sent by the computing unit to the matching data temperature partition or hot replica partition based on the load of each computing task in the computing module accessing hot expert parameters. The copy recycling circuit 2304 is used to continuously monitor the route hit frequency of each hotspot expert computing unit based on the expert routing information table received in real time. When the route hit frequency of the target hotspot expert computing unit is lower than the preset hotspot expert threshold, the hotspot expert parameter copy of the target hotspot expert computing unit in the hotspot copy partition is recycled to release the space of the hotspot copy partition.

[0043] The preset hotspot expert threshold can refer to a hardware-configurable threshold preset in the hotspot identification counter 2301. This threshold is used to compare the route hit count of each expert calculation unit with the actual route hit count to determine if the expert possesses hotspot characteristics. When the route hit count exceeds this threshold, the corresponding expert calculation unit can be identified as a hotspot expert calculation unit. The target hotspot expert calculation unit can refer to a specific expert calculation unit that has already been identified as a hotspot expert calculation unit, and whose route hit frequency is being continuously monitored by the copy recycling circuit 2304.

[0044] Specifically, the hotspot expert identification unit 230 may include a hotspot identification counter 2301, a hardware direct memory access (HDMA) transport circuit 2302, a replica scheduling circuit 2303, and a replica recycling circuit 2304. The hotspot identification counter 2301 receives an expert routing information table sent in real time by the hybrid expert routing unit and monitors the route hit count of each expert computing unit recorded in the information table. When the hotspot identification counter 2301 determines that the route hit count of any expert computing unit exceeds a preset hotspot expert threshold, it identifies that expert computing unit as a hotspot expert computing unit and transmits its expert identifier to the hardware direct memory access (HDMA) transport circuit 2302. Upon receiving the expert identifier from the hotspot identification counter 2301, the hardware direct memory access (HDMA) transport circuit 2302 queries the expert address table based on the expert identifier to obtain the storage addresses of the data blocks required by the hotspot expert computing unit in the pre-storage partition. Simultaneously, it queries the partition mapping information provided by the data hierarchical management unit 220 to locate the storage location of each data block in each data temperature partition. The hardware direct memory access (HMI) transport circuit 2302 retrieves data blocks belonging to the hotspot expert computing unit from each data temperature partition based on the query results, generates a complete copy of the hotspot expert parameters corresponding to the hotspot expert computing unit, and saves the hotspot expert parameter copy to the hotspot copy partition. The copy scheduling circuit 2303 dynamically schedules the hotspot expert parameter access requests sent by each computing unit to the matching data temperature partition or hotspot copy partition based on the load of each computing task accessing the hotspot expert parameters in the computing module, enabling parallel reading and load distribution of the same hotspot expert parameter copy by multiple computing units. The copy recycling circuit 2304 continuously receives the expert routing information table sent in real time by the hybrid expert routing unit and continuously monitors the route hit frequency of each hotspot expert computing unit based on this information table. When the copy recycling circuit 2304 determines that the route hit frequency of the target hotspot expert computing unit is lower than a preset hotspot expert threshold, it reclaims the hotspot expert parameter copy corresponding to the target hotspot expert computing unit from the hotspot copy partition, releasing the storage space of the hotspot copy partition.

[0045] The data recycling unit 240 includes: a window configuration register 2401, a write timestamp recording circuit 2402, an aging comparator 2403, and a background recycling pipeline circuit 2404; Window configuration register 2401 is used to configure the sliding window length corresponding to each recyclable partition; The write timestamp recording circuit 2402 is used to record a global clock-synchronized write timestamp for any data block when it is written to a matching reclaimable partition. The aging comparator 2403 is used to calculate the time storage span between the current system time and the write timestamp of each data block in each recyclable partition in real time, and to identify data blocks whose time storage span exceeds the length of the matching sliding window as aging data, and to add an aging label to the aging data identified in real time. The background recycling pipeline circuit 2404 is used to clear the aging data in the corresponding recyclable partition according to the aging identifier, so as to free up space in each recyclable partition.

[0046] The sliding window length can refer to the hardware-adjustable duration parameter configured for each recyclable partition, which can be used to limit the upper limit of the time that data is allowed to reside in the corresponding partition. The aging flag can refer to the hardware tag signal added to the data block after it is determined to be aging data. This tag signal is associated with the storage address of the data block and can be used to identify the target data block that needs to be reclaimed.

[0047] Specifically, the data recycling unit 240 may include a window configuration register 2401, a write timestamp recording circuit 2402, an aging comparator 2403, and a background recycling pipeline circuit 2404. The window configuration register 2401 is used to configure the corresponding sliding window length for each of the hot data partitions, warm data partitions, or hot copy partitions in the recyclable partitions. The sliding window length of each recyclable partition is independently adjustable to adapt to the differences in timeliness requirements of different data types. The write timestamp recording circuit 2402 is used to record a write timestamp synchronized with the chip's global clock for any data block when it is written to a matching recyclable partition, serving as the timing reference for subsequent aging determination. The aging comparator 2403 is used to calculate the time storage span between the current system time and the write timestamps of each data block in each recyclable partition in real time, and compare this time storage span with the sliding window length matched to the corresponding partition. When the time storage span exceeds the sliding window length, the aging comparator 2403 identifies the data block as aging data and adds an aging flag to the real-time identified aging data. The background recycling pipeline circuit 2404 can be used to clear the aged data in the corresponding recyclable partition during the idle clock cycle of the inference computing pipeline, based on the aging flag added to each data block by the aging comparator 2403, so as to free up the storage space of each recyclable partition.

[0048] Optionally, the data recovery unit 240 may also include a multimodal adaptation circuit 2405; The multimodal adaptation circuit 2405 is used to set independent sliding window lengths for temperature data blocks of various modalities and to switch the recycling strategies of various modal temperature data through a hardware selector; wherein, the modalities of the temperature data blocks include: image, voice and text.

[0049] Specifically, the multimodal adaptation circuit 2405 in the data recycling unit 240 can configure independent window configuration parameters for warm data blocks of image modality, voice modality, and text modality, so that each modality of warm data block resides in the same recyclable partition and follows its own independent timeliness judgment criteria without affecting each other. The multimodal adaptation circuit 2405 realizes dynamic switching of various modal warm data recycling strategies through a hardware selector. When the data recycling unit 240 performs aging judgment on warm data blocks of different modalities, the multimodal adaptation circuit 2405 can select the window configuration parameters corresponding to the modality of the current warm data block through the hardware selector, and apply the corresponding recycling strategy to the sliding window aging judgment process of the warm data block. Through the above-mentioned modality-independent window configuration and hardware selector switching mechanism, the differentiated configuration of recycling strategies for various modal warm data blocks avoids the mutual crowding of storage space by different modal data due to different timeliness requirements.

[0050] The monitoring and protection unit 250 includes: an occupancy counter 2501, a threshold comparison circuit 2502, an adaptive throttling circuit 2503, and an emergency protection circuit 2504; The occupancy counter 2501 is used to calculate the real-time space occupancy rate of each recyclable partition and output the occupancy rate signal corresponding to each recyclable partition to the threshold comparison circuit 2502. The threshold comparison circuit 2502 is used to compare each occupancy rate signal with a preset occupancy rate threshold for each recyclable zone, identify recyclable zones whose occupancy rate signals reach a preset warning ratio of the preset occupancy rate threshold but do not exceed the preset occupancy rate threshold as first-class over-limit zones, and generate a warning signal matching the first-class over-limit zones and transmit it to the adaptive throttling circuit 2503; and identify recyclable zones whose occupancy rate signals exceed the preset occupancy rate threshold as second-class over-limit zones, and generate an overflow protection signal matching the second-class over-limit zones and transmit it to the emergency protection circuit 2504; The adaptive throttling circuit 2503 is used to increase the compression storage parameters of the data to be stored in the first type of over-limit partition and / or tighten the sliding window length in the data recycling unit 240 corresponding to the first type of over-limit partition after receiving the warning signal. Among them, the data blocks of the hot data area corresponding to the warning signal are preferentially migrated to the warm data partition. The emergency protection circuit 2504 is used to reclaim the second type of over-limit partition according to the time storage span of each data block stored in the second type of over-limit partition after receiving the overflow protection signal, and according to the preset reclamation rules. Among them, the data being called in the hot data partition is prohibited from being reclaimed.

[0051] The preset occupancy threshold refers to a hardware-configurable critical value preset for each reclaimable partition. This threshold is used to compare the real-time space occupancy of each partition to determine its saturation level. This threshold can be configured independently for each partition type. The preset warning ratio refers to a preset ratio coefficient relative to the preset occupancy threshold (e.g., 80%, 90%). When the real-time space occupancy of a reclaimable partition reaches this ratio of the preset occupancy threshold, a warning signal is triggered. The first type of over-limit partition refers to a reclaimable partition whose real-time space occupancy reaches the preset warning ratio of the preset occupancy threshold but has not yet exceeded the preset occupancy threshold. This type of partition is in a state of space shortage but still operational, requiring a warning to be triggered for early intervention and adjustment. The warning signal is a warning signal generated after identifying the first type of over-limit partition. It can be used to notify the adaptive throttling circuit 2503 to initiate adjustment operations on the corresponding partition in advance to prevent further space deterioration. The second type of over-limit partition refers to a reclaimable partition whose real-time space occupancy exceeds the preset occupancy threshold. This type of partition is in a dangerous state of space saturation and requires immediate emergency reclamation. Overflow protection signal can refer to an emergency trigger signal generated after identifying a second type of over-limit partition. It can be used to notify the emergency protection circuit 2504 to immediately perform recycling processing on the corresponding partition to prevent storage overflow.

[0052] Specifically, the monitoring and protection unit 250 may include an occupancy counter 2501, a threshold comparison circuit 2502, an adaptive throttling circuit 2503, and an emergency protection circuit 2504. The occupancy counter 2501 calculates the real-time space occupancy rate of each recyclable partition (including hot data partitions, warm data partitions, and hotspot copy partitions) and outputs an occupancy rate signal corresponding to each recyclable partition to the threshold comparison circuit 2502. The threshold comparison circuit 2502 compares the received occupancy rate signals with the preset occupancy rate thresholds for each recyclable partition and performs a two-level judgment based on the comparison results. In the first-level judgment, when the real-time space occupancy rate of a recyclable partition reaches a preset warning ratio of the preset occupancy rate threshold but has not yet exceeded the preset occupancy rate threshold, the threshold comparison circuit 2502 can identify the recyclable partition as a first-class over-limit partition and generate a warning signal matching the first-class over-limit partition, which is then transmitted to the adaptive throttling circuit 2503. In the second-level determination, when the real-time space occupancy rate of a recyclable partition exceeds a preset occupancy rate threshold, the threshold comparison circuit 2502 can identify the recyclable partition as a second-type over-limit partition and generate an overflow protection signal matching the second-type over-limit partition, which is then transmitted to the emergency protection circuit 2504. After receiving the warning signal from the threshold comparison circuit 2502, the adaptive throttling circuit 2503 performs throttling adjustment for the first-type over-limit partition. This can be achieved by increasing the compression storage parameters of the data to be stored in the first-type over-limit partition (e.g., increasing the compression ratio) to reduce the space occupied by newly written data. It can also tighten the sliding window length in the data recycling unit 240 corresponding to the first-type over-limit partition, accelerating the determination and recycling of aging data in that partition, thereby releasing storage space in advance. Specifically, for data blocks in the hot data partition corresponding to the warning signal, they can be preferentially migrated to the warm data partition to reduce the space occupancy pressure of the hot data partition. After receiving the overflow protection signal from the threshold comparison circuit 2502, the emergency protection circuit 2504 immediately performs emergency recycling processing for the second-type over-limit partition. The emergency protection circuit 2504 performs forced reclamation of the second-type over-limit partition based on the time storage span of each data block stored in the partition (i.e., the difference between the current system time and the data block's write timestamp), according to preset reclamation rules (such as prioritizing the reclamation of the data block with the longest time storage span). This quickly releases storage space and reduces the partition space occupancy rate to below a preset occupancy rate threshold. Specifically, data blocks in the hot data partition that are currently being used by the computation module are protected and prohibited from reclamation to ensure that the current inference task is not interrupted.

[0053] Optionally, the hierarchical storage system also includes: a timing coordination control unit 260, wherein the timing coordination control unit 260 is connected to the AFD coordination interface module 270 and the computing module 280 respectively; The timing coordination control unit 260 is used to receive prefetch trigger signals and pipeline timing signals, control the timing of data prefetching, transporting and storage scheduling operations, and synchronize data placement with the computing pipeline.

[0054] The prefetch trigger signal can be a hardware trigger signal sent from the AFD coordination interface module 270 or the hybrid expert routing unit to the timing coordination control unit 260. This signal indicates the data block information (such as the expert identifier and corresponding weight data address to be hit) that will be invoked in subsequent inference steps, thus initiating the data transfer operation in advance. The pipeline timing signal can be a timing indication signal sent from the AFD coordination interface module 270 to the timing coordination control unit 260. This signal reflects the current stage position of the AFD inference pipeline (such as the attention calculation stage, feedforward network calculation stage, or MoE routing stage) and the timing alignment boundaries between stages.

[0055] Specifically, the hierarchical storage system may also include a timing coordination control unit 260, which is connected to both the AFD coordination interface module 270 and the computing module 280. The timing coordination control unit 260 receives prefetch trigger signals from the AFD coordination interface module 270 or the hybrid expert routing unit, as well as pipeline timing signals from the AFD coordination interface module 270. Based on the prefetch trigger signals, the timing coordination control unit 260 determines the information of the data block to be invoked and obtains the current stage position and timing boundaries of each stage of the AFD inference pipeline based on the pipeline timing signals. According to these two signals, the timing coordination control unit 260 controls the initiation timing of data prefetching and data storage scheduling operations in the hierarchical storage module, ensuring that the prefetched data is precisely located in the corresponding storage partition before the computing module 280 initiates an access request. This achieves synchronous matching between the data placement timing and the AFD inference pipeline computing timing, avoiding data waiting or computational idleness.

[0056] Optionally, the timing coordination control unit 260 includes: a prefetch control circuit 2601, a prefetch direct memory access transport circuit 2602, a hardware synchronizer 2603, and a flow rate adaptation circuit 2604; The prefetch control circuit 2601 is used to receive the data input timing signal sent by the AFD cooperative interface module 270 and the expert prediction signal sent by the hybrid expert routing unit in the calculation module 280, and generate a prefetch instruction to be transmitted to the prefetch direct memory access transport circuit 2602. The prefetch direct memory access transport circuit 2602 is used to prefetch the corresponding expert data copy to the hot copy partition of the second-level cache 2902 according to the received prefetch instruction, and preload the target hot expert data to be called from the hot copy partition into the first-level cache 2901. Hardware synchronizer 2603 is used to align the storage scheduling timing with the pipeline timing of AFD coordination interface module 270 and compute module 280; The flow rate adaptation circuit 2604 is used to monitor the data input timing signal of the AFD collaborative interface module 270 in real time and dynamically adjust the data write timing and prefetch frequency of the first-level cache 2901.

[0057] The data input timing signal can refer to the timing indication information sent by the AFD collaborative interface module 270 to the prefetch control circuit 2601 and the flow rate adaptation circuit 2604, reflecting the input rate and timing of the attention calculation unit sending activation value data to the hierarchical storage system. The expert prediction signal can refer to the prediction indication information sent by the hybrid expert routing unit in the calculation module 280 to the prefetch control circuit 2601, which can be used to indicate in advance the storage location of the hybrid expert identifier and corresponding weight data that may be called in the next inference step. The expert data copy can refer to the hotspot expert parameter copy stored in the hotspot copy partition of the secondary cache 2902. The target hotspot expert data can refer to the specific hotspot expert parameter data that the prefetch direct memory access transport circuit 2602 reads from the hotspot copy partition of the secondary cache 2902 according to the prefetch instruction and prepares to load into the primary cache 2901; this data is the data block required by the expert calculation unit that will be called by the current inference step.

[0058] Specifically, the timing coordination control unit 260 may include a prefetch control circuit 2601, a prefetch direct memory access transport circuit 2602, a hardware synchronizer 2603, and a flow rate adaptation circuit 2604. The prefetch control circuit 2601 receives data input timing signals sent by the AFD coordination interface module 270 and expert prediction signals sent by the hybrid expert routing unit in the computing module 280. Based on these two signals, the prefetch control circuit 2601 comprehensively judges the expert data information to be called and its required placement time node, generates a prefetch instruction, and transmits it to the prefetch direct memory access transport circuit 2602. According to the received prefetch instruction, the prefetch direct memory access transport circuit 2602 first prefetches the corresponding expert data copy from the cold or warm data partition to the hot copy partition of the secondary cache 2902, and then preloads the target hot expert data to be called by the current inference step from the hot copy partition of the secondary cache 2902 into the primary cache 2901, ensuring that the data is accurately located at the lowest latency storage level before the computing module 280 initiates an access request. Hardware synchronizer 2603 can be used to perform hardware-level synchronization and alignment of the storage scheduling timing in the hierarchical storage system with the data transmission timing of the AFD coordination interface module 270 and the computation pipeline timing of the computation module 280, ensuring that storage operations and computation operations are precisely matched at each timing boundary. Flow rate adaptation circuit 2604 can monitor the data input timing signal sent by the AFD coordination interface module 270 in real time, obtain the input rate information of the activation value data, and dynamically adjust the data write timing and prefetch frequency of the first-level cache 2901 accordingly, so that the data write speed matches the data input rate, avoiding data accumulation due to excessively fast writing or idle computation units due to excessively slow writing.

[0059] Optionally, the tiered storage module 290 is also connected to the hardware compression and decompression engine 2010.

[0060] The data classification and control unit 220 is specifically used to: output a matching compression ratio control signal to the hardware compression and decompression engine 2010 according to the tag type of the received data tag, so that the hardware compression and decompression engine 2010 can copy the target data block to the matching data temperature partition for compression and storage according to the compression storage parameters matched with the compression ratio control signal, so as to realize differentiated compression and storage of data blocks of different tag types.

[0061] Specifically, the hierarchical storage module 290 can also be connected to the hardware compression and decompression engine 2010. The data hierarchical management unit 220, upon receiving a data tag from the data tag generation unit 210, determines the compression storage parameters matching the current tag type (cold data tag, warm data tag, or hot data tag) based on the tag type, and outputs a corresponding compression ratio control signal to the connected hardware compression and decompression engine 2010. Upon receiving the compression ratio control signal, the hardware compression and decompression engine 2010, based on the compression storage parameters matching the compression ratio control signal, performs a compression storage operation and copies the compressed data corresponding to the target data block to the matching data temperature partition. Through this mechanism, data blocks of different tag types can use differentiated compression storage parameters when written to their respective matching data temperature partitions. For example, cold data can use a high compression ratio to save storage space, while hot data can use a low compression ratio or no compression to ensure the lowest read latency, thereby achieving a balance between storage efficiency and access latency optimization.

[0062] The technical solution of this invention is a hierarchical storage system for large-model inference acceleration chips, including a hierarchical storage module, a data tag generation unit, a data hierarchical management and control unit, a hotspot expert identification unit, a data recycling unit, and a monitoring and protection unit. The hierarchical storage module includes a level 1 cache and a level 2 cache. The hierarchical storage module is connected to an external AFD collaboration interface module and is used to cache data blocks sent in real time by the AFD collaboration interface module in a pre-storage partition. The hotspot expert identification unit is connected to a computing module located outside the system. The computing module includes a hybrid expert routing unit and multiple expert computing units. The data tag generation unit is used to generate matching data tags by access frequency statistics and four-dimensional feature acquisition when a target data block in the pre-storage partition meets the data copy conditions. Furthermore, when a new data block is detected in the pre-storage partition, it records the expert computing unit using the new data block and the storage address of the new data block in the pre-storage partition in the expert address table. The data classification and control unit determines the compressed storage address of the target data block in the matching data temperature partition based on the tag type of the received data tags through hardware lookup table address mapping and completes the copying. Simultaneously, it provides partition mapping information to the hotspot expert identification unit and the data recycling unit. Data temperature partitions include cold data partitions, hot data partitions, and warm data partitions. The hotspot expert identification unit identifies hotspot expert computing units based on the expert routing information table sent in real-time by the hybrid expert routing unit. It retrieves data blocks belonging to each hotspot expert computing unit from each data temperature partition through hardware direct memory access (HDM) transfer circuits, generates hotspot expert parameter copies, saves them to the hotspot copy partition, and uses a copy scheduling circuit to distribute the load across multiple computing units. The data recycling unit identifies aging data based on the write timestamps and corresponding sliding window lengths of data blocks in each recyclable partition. It then performs seamless recycling of aging data through a background recycling pipeline to free up space in each recyclable partition. The monitoring and protection unit identifies over-limit partitions based on the real-time space occupancy rate of each recyclable partition. It adjusts the aging data identification conditions and compression storage parameters of the over-limit partitions through an adaptive throttling circuit, and triggers emergency recycling of the over-limit partitions via an emergency protection circuit. Hot data partitions, warm data partitions, and hotspot replica partitions are recyclable partitions. In addition, the hierarchical storage system includes a timing coordination control unit, which receives prefetch trigger signals and pipeline timing signals, controlling the timing of data prefetching, handling, and storage scheduling operations to synchronize data placement with the computing pipeline. The hierarchical storage module is also connected to a hardware compression and decompression engine. The data hierarchical management unit outputs matching compression ratio control signals based on different tag types, achieving differentiated compression storage for data blocks of different tag types.This technical solution eliminates storage bandwidth contention and access conflicts through physical partitioning and isolation in the data storage scheduling process, reduces inference latency jitter, resolves storage bandwidth bottlenecks through dynamic replication of hotspot experts, improves computing power utilization and inference throughput, and prevents on-chip storage overflow through hardware sliding aging and recycling.

[0063] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0064] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A hierarchical storage system for large-model inference acceleration chips, characterized in that, include: The system includes a hierarchical storage module, a data tag generation unit, a data hierarchical management and control unit, a hotspot expert identification unit, a data recycling unit, and a monitoring and protection unit. The hierarchical storage module includes a first-level cache and a second-level cache. The hierarchical storage module connects to the external attention and feedforward network separation architecture AFD cooperative interface module to cache data blocks sent in real time by the AFD cooperative interface module in the pre-storage partition; The hotspot expert identification unit is connected to a computing module located outside the system. The computing module includes a hybrid expert routing unit and multiple expert computing units. The data tag generation unit is used to generate a data tag matching the target data block and send it to the data hierarchical management and control unit when the target data block in the pre-storage partition meets the data copy conditions. In addition, when a new data block is detected in the pre-storage partition, the expert computing unit that uses the new data block and the storage address of the new data block in the pre-storage partition are recorded in the expert address table; The data classification and control unit is used to copy the target data block to the matching data temperature partition for compressed storage according to the tag type of the received data tag, and to provide the partition mapping information of the target data block from the pre-storage partition to the matching data temperature partition to the hot spot expert identification unit and the data recycling unit; wherein, the data temperature partition includes cold data partition, hot data partition and warm data partition; The hotspot expert identification unit is used to identify the hotspot expert computing units in the computing module based on the expert routing information table sent in real time by the hybrid expert routing unit, and to generate hotspot expert parameter copies by obtaining each data block belonging to the hotspot expert computing unit from each data temperature partition according to the expert address table and the mapping information of each partition, and saving them to the hotspot copy partition; among them, the hot data partition, the temperature data partition and the hotspot copy partition are recyclable partitions. The data recycling unit is used to identify aging data in each recyclable partition based on the write timestamp of each recyclable data block in each recyclable partition and the aging data identification conditions of each recyclable partition, and to recycle the aging data in the background according to the mapping information of each partition, so as to release the space of each recyclable partition. The monitoring and protection unit is used to identify over-limit partitions based on the real-time space occupancy rate of each recyclable partition, adjust the aging data identification conditions and / or compression storage parameters of the over-limit partitions, and trigger emergency recycling for the over-limit partitions.

2. The hierarchical storage system according to claim 1, characterized in that, The types of data tags include: cold data tags that match cold data partitions, warm data tags that match warm data partitions, and hot data tags that match hot data partitions; Pre-storage partitions and cold data partitions are set in the second-level cache, while hot data partitions, hot copy partitions, and warm data partitions are set in the first-level cache or the second-level cache.

3. The hierarchical storage system according to claim 1, characterized in that, The data tag generation unit includes: an access frequency statistics circuit, a four-dimensional feature acquisition circuit, a tag generation logic circuit, and a threshold configuration circuit; The access frequency statistics circuit is used to start timing after any data block is stored in the pre-storage partition, and when the timing reaches a preset fixed time window, determine that the data block is the target data block that meets the data copying conditions, and output the access frequency signal according to the number of accesses to the target data block counted by the hardware timer in the fixed time window; The four-dimensional feature acquisition circuit is used to simultaneously acquire the service attribute signal, life cycle signal and pipeline timing signal of the target data block when the access frequency signal of the target data block is received, and form a four-dimensional feature signal with the access frequency signal, and send the four-dimensional feature signal to the tag generation logic circuit. The tag generation logic circuit is used to generate a data tag that matches the target data block based on the four-dimensional feature signal of the target data block and at least one pre-set access frequency threshold. A threshold configuration circuit is used to configure at least one access frequency threshold that matches the fixed time window according to the needs of the computing task.

4. The hierarchical storage system according to claim 1, characterized in that, The data hierarchical management and control unit includes: a hardware lookup table address mapping circuit, a partition isolation gating circuit, a cold data latching circuit, and an address verification circuit; The hardware lookup table address mapping circuit is used to obtain the matching data temperature partition according to the tag type of the received data tag, and determine the compressed storage address of the target data block in the matching data temperature partition according to the original storage address of the target data block in the pre-storage partition; after establishing the partition mapping information between the original storage address and the compressed storage address, the target data block is copied to the compressed storage address; The partition isolation gating circuit is used to set an independent hardware gating gate for each data temperature partition, allowing access to the storage address of the corresponding data temperature partition only based on the data tag; The cold data latch circuit is used to latch the address range of the cold data partition after the hierarchical storage module is powered on and initialized, to block the replacement and reclamation instructions for the cold data partition, and to perform the corresponding address range latching operation for newly added cold data. The address verification circuit is used to perform real-time verification of the original storage address of the target data block stored in the compressed storage address based on the constructed partition mapping information.

5. The hierarchical storage system according to claim 1, characterized in that, The hotspot expert identification unit includes: a hotspot identification counter, a hardware direct memory access transport circuit, a replica scheduling circuit, and a replica recycling circuit; The hotspot identification counter is used to obtain the number of route hits of each expert computing unit based on the expert routing information table sent in real time by the hybrid expert routing unit. When it is determined that the number of route hits of any expert computing unit exceeds the preset hotspot expert threshold, the expert computing unit is identified as a hotspot expert computing unit, and the expert identifier of the expert computing unit is transmitted to the hardware direct memory access transfer circuit. The hardware direct memory access transfer circuit is used to query the expert address table and the mapping information of each partition according to the received expert identifier, and to obtain the data blocks belonging to the hot spot expert calculation unit from each data temperature partition according to the query results to generate hot spot expert parameter copies and save them to the hot spot copy partition. The replica scheduling circuit is used to dynamically schedule the access requests for hot expert parameters sent by the computing unit to the matching data temperature partition or hot replica partition based on the load of each computing task in the computing module accessing hot expert parameters. The copy recycling circuit is used to continuously monitor the route hit frequency of each hotspot expert computing unit based on the real-time received expert routing information table. When the route hit frequency of the target hotspot expert computing unit is lower than the preset hotspot expert threshold, the hotspot expert parameter copy of the target hotspot expert computing unit in the hotspot copy partition is recycled to release the space of the hotspot copy partition.

6. The hierarchical storage system according to claim 1, characterized in that, The data recycling unit includes: a window configuration register, a write timestamp recording circuit, an aging comparator, and a background recycling pipeline circuit. The window configuration register is used to configure the sliding window length corresponding to each recyclable partition. A write timestamp recording circuit is used to record a global clock-synchronized write timestamp for any data block when it is written to a matching reclaimable partition. The aging comparator is used to calculate the time storage span between the current system time and the write timestamp of each data block in each recyclable partition in real time, and to identify data blocks whose time storage span exceeds the length of the matching sliding window as aging data, and to add an aging label to the aging data identified in real time. The background recycling pipeline circuit is used to clear the aging data in the corresponding recyclable partition according to the aging indicator, so as to free up space in each recyclable partition.

7. The hierarchical storage system according to claim 6, characterized in that, The data recycling unit also includes a multimodal adaptation circuit; The multimodal adaptation circuit is used to set independent sliding window lengths for temperature data blocks of various modalities and to switch the recycling strategies of various modal temperature data through a hardware selector; wherein, the modalities of the temperature data blocks include: image, voice and text.

8. The hierarchical storage system according to claim 6, characterized in that, The monitoring and protection unit includes: a occupancy counter, a threshold comparison circuit, an adaptive throttling circuit, and an emergency protection circuit; The occupancy counter is used to calculate the real-time space occupancy rate of each recyclable partition and output the occupancy rate signal corresponding to each recyclable partition to the threshold comparison circuit. A threshold comparison circuit is used to compare each occupancy rate signal with a preset occupancy rate threshold for each recyclable zone, identify recyclable zones whose occupancy rate signals reach a preset warning ratio of the preset occupancy rate threshold but do not exceed the preset occupancy rate threshold as first-class over-limit zones, and generate a warning signal matching the first-class over-limit zones and transmit it to the adaptive throttling circuit; and identify recyclable zones whose occupancy rate signals exceed the preset occupancy rate threshold as second-class over-limit zones, and generate an overflow protection signal matching the second-class over-limit zones and transmit it to the emergency protection circuit. An adaptive throttling circuit is used to increase the compression storage parameters of the data to be stored in the first type of over-limit partition and / or tighten the sliding window length in the data recycling unit corresponding to the first type of over-limit partition after receiving the warning signal. Among them, the data blocks of the hot data area corresponding to the warning signal are preferentially migrated to the warm data partition. The emergency protection circuit is used to reclaim the second type of over-limit partition according to the time storage span of each data block stored in the second type of over-limit partition after receiving the overflow protection signal, and according to the preset reclamation rules. Among them, the data being accessed in the hot data partition is prohibited from being reclaimed.

9. The hierarchical storage system according to any one of claims 1-8, characterized in that, The hierarchical storage system also includes a timing coordination control unit, wherein the timing coordination control unit is connected to the AFD coordination interface module and the computing module respectively; The timing coordination control unit is used to receive prefetch trigger signals and pipeline timing signals, control the timing of data prefetching, transporting and storage scheduling operations, and synchronize data placement with the computing pipeline.

10. The hierarchical storage system according to claim 9, characterized in that, The timing coordination control unit includes: a prefetch control circuit, a prefetch direct memory access transport circuit, a hardware synchronizer, and a flow rate adaptation circuit. The prefetch control circuit is used to receive the data input timing signal sent by the AFD cooperative interface module and the expert prediction signal sent by the hybrid expert routing unit in the computing module, and generate a prefetch instruction to be transmitted to the prefetch direct memory access transport circuit. The prefetch direct memory access transport circuit is used to prefetch the corresponding expert data copy to the hot copy partition of the second-level cache according to the received prefetch instruction, and preload the target hot expert data to be called from the hot copy partition into the first-level cache. Hardware synchronizers are used to align the storage scheduling timing with the pipeline timing of the AFD coordinating interface module and the compute module; The flow rate adaptation circuit is used to monitor the data input timing signals of the AFD collaborative interface module in real time and dynamically adjust the data write timing and prefetch frequency of the first-level cache.

11. The hierarchical storage system according to any one of claims 1-8, characterized in that, The hierarchical storage module is also connected to a hardware compression and decompression engine; The data classification and control unit is specifically used to: output a matching compression ratio control signal to the hardware compression and decompression engine according to the tag type of the received data tag, so that the hardware compression and decompression engine can copy the target data block to the matching data temperature partition for compression and storage according to the compression storage parameters matched with the compression ratio control signal, so as to achieve differentiated compression and storage of data blocks of different tag types.

Images