A large model inference acceleration chip and a large model inference system

Abstract

The application discloses a large model reasoning acceleration chip and a large model reasoning system, wherein an AFD cooperative interface module receives data information of a current calculation task calculated by an attention cluster in real time, and respectively sends the data information to an on-chip hierarchical storage module and a scheduling and control module. The scheduling and control module generates a calculation instruction and sends the calculation instruction to a calculation module. The calculation module performs a calculation task according to data information read from the on-chip hierarchical storage module, sends a coprocessing instruction to a multi-modal expansion module according to a task type, and feeds back a calculation result to the AFD cooperative interface module. The multi-modal expansion module acquires data information corresponding to the current calculation task from the on-chip hierarchical storage module according to the coprocessing instruction sent by the calculation module, performs a calculation task, and returns a calculation result to the calculation module. The technical scheme improves on-chip storage utilization, reduces data interaction delay, and enhances chip multi-modal reasoning adaptation capability and task scheduling flexibility.

Description

Technical Field

[0001] This invention relates to the field of digital integrated circuit technology, and in particular to a large model inference acceleration chip and a large model inference system. Background Technology

[0002] With the widespread adoption of large-scale Mixed-of-Experts (MoE) and multimodal large models, high-concurrency, low-latency inference has become a necessity. The industry has proposed the AFD (Attention-Feedforward Disaggregation) architecture, which splits the Transformer decoding end into independent units for attention, feedforward networks, and mixed-of-experts. This relies on neural network processing units to achieve resource decoupling and expansion. The technical approach involves three layers of optimization: pre-filling-decoding separation, mixed-expert attention separation, and fully asynchronous data flow. Simultaneously, dedicated streaming inference chips have emerged, using compilers to coordinate hardware and software collaboration, and employing dedicated computing units and on-chip high-speed storage to alleviate the memory wall problem.

[0003] Current AFD (Automatic Detection and Decoding) methods employ PD (Prefill-Decode Disaggregation) in the first stage, separating the prefilling of the key-value cache (KV Cache) and the decoding of tokens onto different hardware to reduce resource contention. The second stage separates the hybrid expert, feedforward network, and attention module to accommodate their different computational requirements. The third stage relies on eliminating global synchronization and hierarchical storage to achieve maximum throughput. Existing heterogeneous solutions use general-purpose graphics hardware to handle prefilling long context tasks, while dedicated inference hardware handles decoding operations, with the chip incorporating matrix, vector, and out-of-order execution units.

[0004] Existing dedicated inference hardware is only compatible with the second stage of AFD and cannot meet the ultra-high concurrency inference requirements of the third stage: the on-chip static random access storage capacity is insufficient to support the large-capacity shared data of hybrid expert models with thousands of experts, and data overflow is prone to increase latency; the latency of the attention-feedforward-attention (AFA, Attention-FFN-Attention) interaction interface does not meet the microsecond-level interaction standard for high-throughput scenarios; there is no dedicated multimodal coding unit, resulting in poor scalability of image-text-speech fusion inference; it only supports static scheduling and lacks dynamic-static hybrid strategies and the ability to dynamically replicate hot experts, resulting in uneven load on hybrid experts, low computing power utilization, and significant latency fluctuations. Summary of the Invention

[0005] This invention provides a large-model inference acceleration chip and a large-model inference system to solve the technical problems of insufficient storage capacity, high latency of module interaction interfaces, weak multimodal inference expansion capabilities, and a single scheduling mechanism in existing dedicated inference hardware.

[0006] According to one aspect of the present invention, a large model inference acceleration chip is provided, comprising: a multimodal extension module, a scheduling and control module, an on-chip hierarchical storage module, a computing module, and an attention and feedforward network separation architecture AFD collaborative interface module; the AFD collaborative interface module is used to connect to an attention cluster;

[0007] The scheduling and control module is bidirectionally interconnected with both the computing module and the AFD collaborative interface module. The on-chip hierarchical storage module, computing module, and multimodal expansion module are bidirectionally interconnected in pairs. A unidirectional data connection is established between the AFD collaborative interface module and the on-chip hierarchical storage module, and a unidirectional data connection is established between the computing module and the AFD collaborative interface module. The on-chip hierarchical storage module includes a primary storage unit and a secondary storage unit, wherein:

[0008] The AFD collaborative interface module is used to receive the activation value, hybrid expert routing table, quantization scaling parameters and feature value sequence of the current computing task calculated in real time by the attention cluster, send them to the matching storage unit in the on-chip hierarchical storage module, and send the current computing task description information sent by the attention cluster to the scheduling and control module.

[0009] The attention cluster is used to calculate activation values, hybrid expert routing tables, and quantization scaling parameters in real time based on the multimodal raw data of the current computing task from external input, and to generate matching description information of the current computing task.

[0010] The scheduling and control module is used to generate computing instructions and send them to the computing module based on the received current computing task description information;

[0011] The computation module is used to respond to computation instructions, and perform hybrid expert route distribution and feedforward network computation for the current computation task based on the activation value, hybrid expert routing table, quantization scaling parameters and feature value sequence read from the on-chip hierarchical storage module. It also issues coprocessing instructions to the multimodal extension module according to the task type of the current computation task, and feeds back the computation results to the AFD cooperative interface module in real time.

[0012] The multimodal extension module is used to obtain the feature value sequence corresponding to the current computing task from the matching storage unit in the on-chip hierarchical storage module according to the coprocessing instructions issued by the computing module, perform feature value sequence calculation, and return the calculation result to the computing module.

[0013] The AFD collaborative interface module is also used to send the real-time received calculation results back to the attention cluster for the attention cluster to perform subsequent attention calculations.

[0014] In another aspect of the present invention, a large model inference system is also provided, comprising: at least one large model inference acceleration chip as described in any embodiment of the present invention and an attention cluster;

[0015] The attention clusters are connected to the AFD collaborative interface modules in the various model inference acceleration chips.

[0016] In the technical solution of this invention embodiment, the AFD collaborative interface module receives the activation value, hybrid expert routing table, quantization scaling parameters, and feature value sequence calculated in real time based on the multimodal raw data of the current computing task output by the attention cluster. It then transmits these data to a matching storage unit within the on-chip hierarchical storage module and forwards the current computing task description information sent by the attention cluster to the scheduling and control module. The scheduling and control module generates a computing instruction based on the received current computing task description information and sends it to the computing module. The computing module responds to the computing instruction, reads the activation value, hybrid expert routing table, quantization scaling parameters, and feature value sequence from the on-chip hierarchical storage module, completes the hybrid expert route distribution and feedforward network calculation corresponding to the current computing task, and simultaneously sends a coprocessing instruction to the multimodal extension module based on the task type of the current computing task. The module also feeds back the calculated results to the AFD collaborative interface module in real time. After receiving the coprocessing instructions from the computing module, the multimodal extension module retrieves the feature value sequence corresponding to the current computing task from the matching storage unit of the on-chip hierarchical storage module, performs feature value sequence calculation, and then sends the calculation result back to the computing module. Finally, the AFD collaboration interface module also sends the received calculation result back to the attention cluster for subsequent attention calculation. This technical solution, through a chip structure combining on-chip hierarchical storage, multimodal data computing hardware unit allocation, and AFD collaboration, significantly improves on-chip storage utilization, reduces data interaction latency between modules, and enhances the chip's multimodal inference adaptability and task scheduling flexibility.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a schematic diagram of the structure of a large model inference acceleration chip according to Embodiment 1 of the present invention;

[0020] Figure 2 This is a schematic diagram of the structure of a large model inference acceleration chip according to Embodiment 2 of the present invention;

[0021] Figure 3 This is a schematic diagram of the structure of a large-scale model inference system provided in Embodiment 3 of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Example 1

[0025] Figure 1 This is a schematic diagram of a large-model inference acceleration chip provided in Embodiment 1 of the present invention. This embodiment is applicable to high-concurrency, ultra-low-latency inference deployments of large-scale hybrid expert models and multimodal large models. Figure 1As shown, the large model inference acceleration chip 100 includes: an AFD collaborative interface module 110 with a separate attention and feedforward network architecture, a scheduling and control module 120, a computing module 130, a multimodal extension module 140, and an on-chip hierarchical storage module 150. The AFD collaborative interface module 110 is used to connect to the attention cluster 160. The scheduling and control module 120 is bidirectionally interconnected with the computing module 130 and the AFD collaborative interface module 110, respectively. The on-chip hierarchical storage module 150, the computing module 130, and the multimodal extension module 140 are bidirectionally interconnected in pairs. A unidirectional data connection is established between the AFD collaborative interface module 110 and the on-chip hierarchical storage module 150, and a unidirectional data connection is established between the computing module 130 and the AFD collaborative interface module 110, respectively. The on-chip hierarchical storage module 150 includes a primary storage unit and a secondary storage unit (not shown in the figure).

[0026] The AFD collaborative interface module 110 is used to receive the activation value, hybrid expert routing table, quantization scaling parameters and feature value sequence of the current computing task calculated in real time by the attention cluster 160, send them to the matching storage unit in the on-chip hierarchical storage module 150, and send the current computing task description information sent by the attention cluster 160 to the scheduling and control module 120.

[0027] The attention cluster 160 is used to calculate activation values, hybrid expert routing tables, and quantization scaling parameters in real time based on the multimodal raw data of the current computing task from external input, and to generate matching description information of the current computing task.

[0028] The activation values ​​refer to the intermediate feature data output by the large model network after activation function operation. These values ​​can be used to carry the effective input features extracted by the large model network, serving as input for subsequent large model computation units and participating in iterative computations of the large model inference. The hybrid expert routing table refers to the hardware addressing mapping data table used by the large model inference acceleration chip to distribute data to various expert hardware computation units. The quantization scaling parameters refer to the hardware constant configuration parameters used for numerical range calibration in the quantization operation path of the large model inference acceleration chip. The feature value sequence refers to the continuous feature data stream transmitted in an ordered manner within the large model inference acceleration chip, available for subsequent hardware units to read; it is the core processing object matched with the computation task. The current computation task description information refers to the task identifier message containing information such as the task type, resource requirements, and data scale of the currently processed computation task. Multimodal raw data refers to computation task data such as text, images, or speech that has not undergone computational processing. The attention cluster refers to a computational group composed of multiple dedicated attention hardware computation units, which can independently carry out the attention mechanism computation of the large model under the attention-feedforward separation architecture, completing pre-inference computation based on multimodal raw data and outputting various intermediate model data.

[0029] It is understandable that the AFD collaborative interface module 110 can be used for data transfer and information distribution between heterogeneous inference clusters. It can receive various types of computational data information, such as activation values, hybrid expert routing tables, quantization scaling parameters, and feature value sequences, generated by the upstream attention cluster 160 after completing multimodal raw data computation, as well as the corresponding current computation task description information. It can also split and write various types of computational data into the corresponding storage units of the on-chip hierarchical storage module 150, and forward the current computation task description information to the scheduling and control module 120, thereby completing the input and transmission operations of data information between the attention cluster 160, the on-chip hierarchical storage module 150, and the scheduling and control module 120.

[0030] The scheduling and control module 120 is used to generate computing instructions and send them to the computing module 130 based on the received current computing task description information.

[0031] It is understandable that after receiving the current computing task description information forwarded by the AFD collaboration interface module 110, the scheduling and control module 120 can parse the current computing task's computing power requirements, data specifications, and operation type, and thereby generate a dedicated computing instruction that matches the hardware resources. The computing instruction is then sent to the computing module 130, which can be understood as the specific computing unit that schedules and processes the current computing task.

[0032] The computing module 130 is used to respond to computing instructions, perform hybrid expert route distribution and feedforward network calculation for the current computing task based on the activation value, hybrid expert routing table, quantization scaling parameters and feature value sequence read from the on-chip hierarchical storage module 150, issue coprocessing instructions to the multimodal extension module 140 according to the task type of the current computing task, and feed back the calculation results to the AFD cooperative interface module 110 in real time.

[0033] Among them, the coprocessing instruction can refer to the control instruction issued to the multimodal extension module 140 to collaboratively complete the processing of the current computing task when the multimodal raw data corresponding to the current computing task is image or voice data.

[0034] Understandably, after receiving the computation instruction from the scheduling and control module 120, the computation module 130 can obtain the storage address of the matching data information required to execute the current computation task in the on-chip hierarchical storage module 150 by parsing the computation instruction. It can then obtain the activation value, hybrid expert routing table, quantization scaling parameters, and feature value sequence corresponding to the current computation task from the corresponding storage address in the on-chip hierarchical storage module 150, complete the hybrid expert routing allocation and feedforward network operation for the current computation task, and output the coprocessing instruction to the multimodal extension module 140 if the task type of the current computation task corresponds to the original data of image or speech. It can also receive the computation result fed back by the multimodal extension module 140, and finally send the computation result obtained after performing the operation on the current computation task back to the AFD cooperative interface module 110.

[0035] The multimodal extension module 140 is used to obtain the feature value sequence corresponding to the current computing task from the matching storage unit in the on-chip hierarchical storage module 150 according to the coprocessing instructions issued by the computing module 130, perform feature value sequence calculation, and return the calculation result to the computing module 130.

[0036] Understandably, when the task type of the current computing task corresponds to the raw data of image or speech, the computing module 130 generates a corresponding coprocessing instruction and sends it to the multimodal extension module 140. The multimodal extension module 140 can parse the received coprocessing instruction to obtain the storage unit in the on-chip hierarchical storage module 150 corresponding to the data information required to execute the current computing task, as well as the storage address of the required data information. It can obtain the feature value sequence corresponding to the current computing task from the matching storage unit, allocate it to the hardware computing unit (not shown in the figure) inside the multimodal extension module 140 for feature value sequence calculation, and return the completed calculation result to the computing module 130.

[0037] The AFD collaborative interface module 110 is also used to send the real-time received calculation results back to the attention cluster 160 so that the attention cluster 160 can perform subsequent attention calculations.

[0038] It is understandable that after the AFD collaborative interface module 110 receives the calculation result obtained by the calculation module 130 in executing the current calculation task, it can send the calculation result back to the attention cluster 160 so that the attention cluster 160 can perform the next stage of attention-related inference operation based on the returned calculation result data, thereby realizing the data loop iteration of the large model inference process for multimodal raw data.

[0039] In the technical solution of this invention embodiment, the AFD collaborative interface module receives the activation value, hybrid expert routing table, quantization scaling parameters, and feature value sequence calculated in real time based on the multimodal raw data of the current computing task output by the attention cluster. It then transmits these data to a matching storage unit within the on-chip hierarchical storage module and forwards the current computing task description information sent by the attention cluster to the scheduling and control module. The scheduling and control module generates a computing instruction based on the received current computing task description information and sends it to the computing module. The computing module responds to the computing instruction, reads the activation value, hybrid expert routing table, quantization scaling parameters, and feature value sequence from the on-chip hierarchical storage module, completes the hybrid expert route distribution and feedforward network calculation corresponding to the current computing task, and simultaneously sends a coprocessing instruction to the multimodal extension module based on the task type of the current computing task. The module also feeds back the calculated results to the AFD collaborative interface module in real time. After receiving the coprocessing instructions from the computing module, the multimodal extension module retrieves the feature value sequence corresponding to the current computing task from the matching storage unit of the on-chip hierarchical storage module, performs feature value sequence calculation, and then sends the calculation result back to the computing module. Finally, the AFD collaboration interface module also sends the received calculation result back to the attention cluster for subsequent attention calculation. This technical solution, through a chip structure combining on-chip hierarchical storage, multimodal data computing hardware unit allocation, and AFD collaboration, significantly improves on-chip storage utilization, reduces data interaction latency between modules, and enhances the chip's multimodal inference adaptability and task scheduling flexibility.

[0040] Example 2

[0041] Figure 2 This is a schematic diagram of a large-model inference acceleration chip provided in Embodiment 2 of the present invention. This embodiment is a further specification based on the above embodiments. Figure 2 As shown, it includes:

[0042] The AFD cooperative interface module 210 specifically includes an attention-to-feedforward network interface 2101, a feedforward network-to-attention interface 2102, a communication direct memory access controller 2103, a ping-pong buffer unit 2104, and an AFD signaling control unit 2105, wherein:

[0043] Attention-to-feedforward network interface 2101 is used to receive activation values, hybrid expert routing tables, quantization scaling parameters and feature value sequences sent by attention cluster 260.

[0044] The feedforward network to attention interface 2102 is used to send the calculation results sent by the calculation module 230 back to the attention cluster 260.

[0045] The communication direct memory access controller 2103 is used to distribute the activation values, hybrid expert routing table, quantization scaling parameters and feature value sequence obtained from the attention-to-feedforward network interface 2101 to the matching storage units in the on-chip hierarchical storage module 220, and to aggregate the calculation results sent in real time by the computing module 230 to the feedforward network to attention interface 2102.

[0046] The ping-pong buffer unit 2104 is used to buffer the data flowing through the attention-to-feedforward network interface and the feedforward network-to-attention interface 2102 to avoid interface data congestion.

[0047] AFD signaling control unit 2105 is used to establish a bidirectional interconnection mechanism between AFD collaborative interface module 210 and attention cluster based on a preset communication protocol.

[0048] It is understandable that the AFD collaborative interface module 210 serves as the interaction hub between the on-chip attention cluster and the computing module 230, the on-chip hierarchical storage module 220, and the scheduling and control module 250. It adopts a dedicated interface design and a high-speed interconnect architecture, and can achieve low-latency transmission and accurate distribution of computational data and bidirectional collaboration of control signaling through the collaborative work of the included functional units.

[0049] Among them, the attention-to-feedforward network interface 2101 is a dedicated receiving path for the output data of the attention cluster. It uses a high-speed differential signal line as the physical transmission medium and can be configured with multiple parallel transmission channels to meet the needs of high-volume data transmission. The single channel and total bandwidth can support the synchronous transmission of activation values, hybrid expert routing tables, quantization scaling parameters and feature value sequences. The transmission delay can be controlled at the microsecond level. Its core function is to receive the real-time output activation values, hybrid expert routing tables, quantization scaling parameters and feature value sequences of the attention cluster, and provide input data information for subsequent data storage and calculation.

[0050] The feedforward network to attention interface 2102 is a dedicated path for transmitting computation results back. It forms a bidirectional data transmission link with the attention to feedforward network interface 2101. It can also adopt a high-speed transmission medium and parallel channel design to ensure the bandwidth requirements of the transmitted data. The transmission delay can be controlled at the microsecond level. Its core function is to transmit the computation results output by the computation module 230 to the attention cluster in real time, supporting the attention cluster to carry out subsequent iterative attention computation.

[0051] The Communication Direct Memory Access Controller 2103 is the core execution unit for data distribution and aggregation. It can use a lock-free access mechanism to autonomously complete data transfer without the intervention of a general-purpose processor, thereby reducing the latency loss caused by processor scheduling. Its core function is to accurately distribute the activation values, hybrid expert routing tables, quantization scaling parameters, and feature value sequences obtained from the attention-to-feedforward network interface 2101 to the matching storage units in the on-chip hierarchical storage module 220 according to the data type. In addition, it can aggregate and organize the calculation results sent in real time by the calculation module 230 and transmit them to the feedforward network-to-attention interface 2102. It can also support the overlapping execution of the calculation process and the communication process, further improving the data transmission efficiency and compressing the overall inference latency.

[0052] The ping-pong buffer unit 2104 can adopt a dual-buffered alternating read-write ping-pong transmission mechanism to provide temporary buffer space for bidirectional interface data. Its core function is to buffer the data flowing through the attention to the feedforward network interface 2101 and the feedforward network to the attention interface 2102. When one side buffer performs data writing, the other side buffer can simultaneously complete data reading. This can avoid interface data blockage caused by differences in processing speeds of upstream and downstream modules and fluctuations in data flow. At the same time, it can realize continuous input and output of data, further smooth the transmission process, reduce transmission jitter, and ensure the stability of data transmission.

[0053] The AFD signaling control unit 2105 is a dedicated unit for controlling signaling interaction. It can use a dedicated signaling communication protocol and low-voltage differential signal lines as the physical medium to reduce signaling transmission delay and meet the requirements of real-time collaborative control. Its core function is to establish a stable bidirectional interconnection mechanism between the AFD collaborative interface module 210 and the attention cluster based on a preset communication protocol. It supports collaborative interaction functions such as dynamic adjustment of computing power ratio, scheduling synchronization, and fault status notification. It can ensure the operational coordination and consistency of the AFD collaborative interface module 210 and the attention cluster during the inference process and realize the precise joint scheduling of the two.

[0054] Furthermore, the on-chip hierarchical storage module 220 also includes a hardware compression and decompression unit 2201, a first-level storage unit 2202 which is an L1 streaming static random access memory, and a second-level storage unit 2203 which is an L2 shared static random access memory, or an L2 shared dynamic random access memory; the first-level storage unit 2202 is configured to use a streaming read / write mode and a write-after-read mechanism to achieve low-latency data reading.

[0055] Correspondingly, the communication direct memory access controller 2103 is specifically used to write the activation values ​​obtained from the attention-to-feedforward network interface 2101 into the first-level storage unit 2202 in batches according to the activation flow order; and to write the hybrid expert routing table, quantization scaling parameters, and various hybrid expert weights used by the calculation module 230 obtained from the attention-to-feedforward network interface 2101 into the second-level storage unit 2203 according to the address mapping rules, so that the calculation module 230 can read the data through a dedicated address line during subsequent calculations.

[0056] The hardware compression and decompression unit 2201 is specifically used to perform streaming hardware compression processing on the long activation value sequence, obtain simplified compressed data and write it to the secondary storage unit 2203, and decompress the target simplified compressed data before sending the target simplified compressed data stored in the secondary storage unit 2203 to the data requester.

[0057] Specifically, the on-chip tiered storage module 220 adopts a tiered storage architecture and is directly interconnected with the computing module 230. It adapts to the access latency and capacity requirements of different types of data through different tiered storage units. At the same time, it integrates hardware compression and decompression unit 2201 to improve storage resource utilization and provide on-chip data storage support for large model inference operations.

[0058] The on-chip hierarchical storage module 220 includes two types of storage entities: a primary storage unit 2202 and a secondary storage unit 2203. It also incorporates a hardware compression and decompression unit 2201. The primary storage unit 2202 can be an L1 streaming static random access memory, configured for streaming read / write mode and equipped with a write-after-read mechanism. Designed for low-latency read scenarios, it can adopt a dual-port read / write architecture, supporting parallel execution of data writing and reading. Data can be read immediately after writing without waiting for cache synchronization, meeting the access requirements of high-real-time data with activation values. The secondary storage unit 2203 can be implemented using either L2 shared static random access memory or L2 shared dynamic random access memory. It can achieve large-capacity storage through stacked three-dimensional storage technology, meeting the storage needs of large models with multiple parameters and long-sequence data. This hierarchical storage is optimized for read-only access and is suitable for storing data with stable access frequency and higher capacity requirements, such as hybrid expert routing tables, quantization scaling parameters, and hybrid expert weights.

[0059] As the execution unit for data writing, the Direct Memory Access Controller 2103 can autonomously complete the directional and classified writing of different types of data without the intervention of a general-purpose processor. For the activation values ​​with the highest real-time requirements, the Direct Memory Access Controller 2103 can write the activation values ​​to the primary storage unit 2202 in batches according to the order of the activation stream. The amount of data written in each batch matches the actual needs of the computing module 230 in a single operation, which can avoid data accumulation in the storage unit or idle operation. At the same time, in conjunction with the streaming read and write characteristics of the primary storage unit 2202, it ensures the continuous and low-latency supply of activation values. For the hybrid expert routing table, quantization scaling parameters, and various hybrid expert weights used by the calculation module 230, the communication direct memory access controller 2103 can write various types of data into the storage blocks corresponding to the secondary storage unit 2203 according to the preset address mapping rules. The hybrid expert routing table and quantization scaling parameters can be stored in a mode of writing once and then reading in a loop. When the calculation module 230 calls the required data, it can quickly locate and read it through a dedicated address line without passing through an intermediate cache. It can also support parallel reading by multiple calculation units, ensuring that there is no bandwidth bottleneck for routing addressing and parameter calling.

[0060] The hardware compression and decompression unit 2201 can employ a hardware-level streaming compression algorithm to perform data compression, storage, and decompression without additional computational performance loss. For long activation value sequences with large data volumes, the hardware compression and decompression unit 2201 can perform streaming hardware compression processing, reducing the data volume while preserving the validity of data computation. The resulting compressed data is then written to the secondary storage unit 2203, effectively increasing the actual usable space of the secondary storage unit 2203. When the target compressed data stored in the secondary storage unit 2203 needs to be sent to the data requester, the hardware compression and decompression unit 2201 can perform real-time decompression processing on the target compressed data before data transmission, restoring it to its original specifications before outputting it, ensuring that the downstream computing module 230 can directly use data that meets the computational requirements.

[0061] Furthermore, the computing module 230 specifically includes a multi-pipeline engine 2301, a hybrid expert model MOE native computing engine 2302, a sparse feedforward network array 2303, and a low-precision computing unit 2304. Among them, the hybrid expert model MOE native computing engine 2302, the sparse feedforward network array 2303, and the low-precision computing unit 2304 all include multiple expert hardware computing units of a set type.

[0062] The expert hardware computing unit in the MOE native computing engine 2302 of the hybrid expert model is used to adapt to the characteristics of the hybrid expert algorithm and complete the complex comparison and nonlinear operation computing tasks required for the selection of top high-scoring experts. The expert hardware computing unit in the sparse feedforward network array 2303 is used to perform the conventional numerical calculation task of basic matrix multiplication. The expert hardware computing unit in the low-precision computing unit 2304 is used to perform low-precision quantization and dequantization tasks.

[0063] The multi-pipeline engine 2301 is used to respond to computation instructions by obtaining the hybrid expert routing table from the secondary storage unit 2203, determining the type and target number of expert hardware computing units to execute the current computation task based on the task characteristics of the current computation task, and selecting the target execution unit from the hybrid expert model MOE native computing engine 2302, sparse feedforward network array 2303 and low-precision computing unit 2304 according to the determined type of expert hardware computing unit, and sending expert hardware computing unit allocation instructions to the target execution unit.

[0064] The target execution unit is used to trigger the activation of the target number of expert hardware computing units according to the target number of expert hardware computing units carried in the expert hardware computing unit allocation instruction issued by the multi-pipeline engine 2301, and to obtain the activation value, quantization scaling parameter and feature value sequence from the primary storage unit 2202 and the secondary storage unit 2203 respectively and distribute them to the activated expert hardware computing units to execute the computing task.

[0065] Here, the expert hardware computing unit can refer to the smallest hardware computing entity integrated within each execution unit of the computing module 230, which performs computing tasks and can be adapted to different types of computing scenarios according to its computing capabilities. The target execution unit can refer to the computing execution entity of the expert hardware computing unit on the computing module 230 selected by the multi-pipeline engine 2301 according to the computing instructions, which is of the type matching the current computing task. It can be the expert hardware computing unit in the hybrid expert model MOE native computing engine 2302, the sparse feedforward network array 2303, or the low-precision computing unit 2304.

[0066] Specifically, the computing module 230 is the core execution unit for on-chip computing tasks. It can be directly connected to the on-chip hierarchical storage module 220 via a dedicated computing architecture. Through the unified scheduling of the three types of computing task execution units by the multi-pipeline engine 2301, hardware-level acceleration of hybrid expert routing distribution, feedforward network operation and quantization operation can be achieved, which can adapt to the computing requirements of high throughput and low latency of large model inference. Among them, the hybrid expert model MOE native computing engine 2302, sparse feedforward network array 2303 and low-precision computing unit 2304 are all equipped with multiple corresponding types of expert hardware computing units.

[0067] The expert hardware computing unit inside the MOE native computing engine 2302 of the hybrid expert model can adapt to the operation characteristics of the hybrid expert algorithm. It can be used to complete the complex comparison and nonlinear operations required for the selection of expert hardware computing units, realize hardware-level route selection acceleration, quickly complete the location of the target execution unit, and provide accurate routing results for the execution of the current computing task.

[0068] The expert hardware computing units inside the sparse feedforward network array 2303 are basic numerical computing execution units, which can be used to perform conventional numerical computing tasks such as basic matrix multiplication. They can support sparse computing mode, activating only the target execution units required for the current task, avoiding computing power redundancy and improving computing power utilization efficiency. In addition, multiple expert hardware computing units can execute feedforward network operations in parallel, adapting to the computing needs of large models and multiple levels.

[0069] The expert hardware computing unit inside the low-precision computing unit 2304 can be used to perform low-precision quantization and dequantization tasks. The hardware natively supports multi-precision quantization, which reduces power consumption and hardware resource usage while ensuring computational accuracy, thus balancing inference accuracy and energy efficiency.

[0070] The multi-pipeline engine 2301 is the core scheduling unit of the computing module 230. It responds to the computing instructions issued by the scheduling and control module 250, and coordinates and manages the allocation and timing of computing resources. It can obtain the hybrid expert routing table from the secondary storage unit 2203 according to the computing instructions, and determine the type and target number of expert hardware computing units required to execute the current task in combination with the task characteristics of the current computing task. Then, according to the determined expert hardware computing unit type, it selects the corresponding target execution unit from the hybrid expert model MOE native computing engine 2302, sparse feedforward network array 2303 and low-precision computing unit 2304, and sends the expert hardware computing unit allocation instruction to the target execution unit. In addition, a multi-level pipelined computing mechanism is adopted, and bubble-free parallel computing is achieved through timing control. The multi-level feedforward network operation can be divided into multiple pipeline segments for cascaded execution. After the operation of the previous pipeline segment is completed, the result can be directly passed to the next pipeline segment without waiting for the entire batch of data to be calculated. At the same time, the storage unit can be coordinated to push the operation data to the computing unit register in advance to avoid the computing unit waiting for data, eliminate pipeline segment idleness, and improve the overall computing efficiency. Furthermore, it supports the parallel activation and synchronous execution of multiple expert hardware computing units, making full use of computing resources in sparse activation scenarios and avoiding the waste of hardware computing resources.

[0071] After receiving the expert hardware computing unit allocation instruction issued by the multi-pipeline engine 2301, the target execution unit triggers the activation of the corresponding number of expert hardware computing units according to the target number of expert hardware computing units carried in the instruction. It can obtain the activation value from the primary storage unit 2202, and obtain the quantization scaling parameters and feature value sequence from the secondary storage unit 2203. It distributes various types of computational data to all activated expert hardware computing units, and each expert unit executes the corresponding computational task in parallel to jointly complete the hybrid expert routing distribution and feedforward network computation of the current computational task.

[0072] Furthermore, the multimodal extension module 240 specifically includes a multimodal streaming unit 2401, a long context acceleration unit 2402, and a shared buffer unit 2403. The multimodal streaming unit 2401 internally includes multiple expert hardware computing units, which are used to perform computational tasks corresponding to multimodal raw data of speech or image data types, wherein:

[0073] The long context acceleration unit 2402 is used to read the feature value sequence and activation value sequence matching the current computing task from the primary storage unit 2202 according to the received coprocessing instructions, and save them to the shared cache unit 2403. During the data saving process, the feature value sequence and activation value sequence that overflow the space of the shared cache unit 2403 read from the primary storage unit 2202 are compressed and saved. In addition, the compressed data is decompressed before the multimodal stream unit 2401 obtains the compressed data from the shared cache unit 2403.

[0074] The multimodal flow unit 2401 is used to trigger the activation of a number of matching hardware computing units according to the coprocessing instructions of the current computing task sent by the computing module 230, and to obtain the matching feature value sequence and activation value sequence from the shared cache unit 2403 and distribute them to the activated hardware computing units to perform feedforward network computing, and return the computing results to the computing module 230.

[0075] Specifically, the multimodal extension module 240 is a functional extension unit of the computing module 230. It is interconnected at high speed with the computing module 230 and the on-chip hierarchical storage module 220. It can be used to adapt to multimodal and long sequence inference scenarios. Through the multimodal stream unit 2401, the long context acceleration unit 2402 and the shared cache unit 2403, it can complete multimodal exclusive operations and efficient processing of long sequence data. The operation process is executed independently of the core computing module 230 and does not occupy the core computing bandwidth. It can expand the chip's multimodal data type adaptation capability without affecting the core inference efficiency.

[0076] The long context acceleration unit 2402 is the core of preprocessing long sequence data. It responds to coprocessing instructions issued by the computing module 230 and performs corresponding operations. Based on the received coprocessing instructions, it can read the feature value sequence and activation value sequence matching the current computing task from the primary storage unit 2202 and transfer the data to the shared cache unit 2403, providing data preparation for subsequent multimodal operations. During the process of saving data to the shared cache unit 2403, if the amount of feature value sequence and activation value sequence read from the primary storage unit 2202 exceeds the available space of the shared cache unit 2403, the overflowing long sequence data will be stream-compressed before being saved. This effectively reduces the storage footprint of long sequence data and adapts to the large-capacity data storage requirements of long context inference scenarios. Before the multimodal stream unit 2401 retrieves compressed data from the shared buffer unit 2403, the long context acceleration unit 2402 will decompress the compressed data in advance to restore it to the original specification of sequence data, ensuring that the multimodal stream unit 2401 can directly read data that meets the operation requirements without introducing additional operation delay due to compression and decompression.

[0077] The multimodal stream unit 2401 is the execution carrier for multimodal-specific computation. It completes the computational operation of the current computational task through the internal multimodal-specific expert hardware computing unit. According to the coprocessing instructions of the current computational task sent by the computing module 230, it triggers the activation of a matching number of expert hardware computing units based on the task computation requirements. It obtains the corresponding feature value sequence and activation value sequence from the shared cache unit 2403, distributes the computational data to all activated expert hardware computing units to perform feedforward network computation, and after completing the multimodal-specific feature computation, returns the computation result to the computing module 230, thus completing the complete multimodal inference process in conjunction with the core computing module 230.

[0078] The shared cache unit 2403 is a high-speed temporary cache carrier inside the multimodal extension module 240. It can be used to temporarily store the feature value sequence, activation value sequence and compressed data read from the on-chip storage by the long context acceleration unit 2402. It provides local high-speed cache support for data interaction between the long context acceleration unit 2402 and the multimodal stream unit 2401, and reduces the transmission latency of cross-module data interaction.

[0079] Furthermore, the scheduling and control module 250 specifically includes: a deterministic execution instruction stream orchestration unit 2501 and a hybrid scheduling unit 2502, wherein:

[0080] The hybrid scheduling unit 2502 is used to call the instruction orchestration tool provided by the deterministic execution instruction stream orchestration unit 2501 according to the received current computing task description information, generate computing instructions containing task types matching the current computing task, and send them to the computing module 230.

[0081] Specifically, the scheduling and control module 250 is the core of global task scheduling and computation instruction management for the large model inference acceleration chip 200. It is responsible for receiving the input current computation task information and generating standardized hardware execution instructions, which are then issued to the computation module 230 to drive the orderly execution of inference operations. The module specifically includes a deterministic execution instruction stream orchestration unit 2501 and a hybrid scheduling unit 2502, which work together to generate and issue computation instructions.

[0082] The hybrid scheduling unit 2502 is the main body for task scheduling execution of the scheduling and control module 250. It is responsible for receiving the current computing task description information from the external input, parsing the core information such as the task type and computing requirements, calling the instruction orchestration tool provided by the deterministic execution instruction stream orchestration unit 2501, generating computing instructions that match the task type of the current computing task, and sending the generated computing instructions to the computing module 230 to trigger the computing module 230 to start the corresponding inference operation.

[0083] The deterministic execution instruction stream orchestration unit 2501 is the underlying support unit for generating computation instructions. It has built-in standardized instruction orchestration rules and instruction templates adapted to different task types. It provides a unified instruction orchestration tool to the outside world. It can respond to the call requirements of the hybrid scheduling unit 2502 and output deterministic computation instructions that conform to the hardware execution specifications based on the task type. This ensures the accuracy of the instructions and the determinism of the execution timing, avoids the fluctuation of computational delay caused by the randomness of instruction scheduling, and supports the stable and low-latency execution of the inference process.

[0084] Furthermore, the multimodal extension module 240 is used to send the real-time load information of each hardware computing unit participating in the feedforward network computing in the multimodal flow unit 2401 to the computing module 230.

[0085] The computing module 230 is used to send the real-time load information of each expert hardware computing unit participating in feedforward network computing in the hybrid expert model MOE native computing engine 2302, sparse feedforward network array 2303, low-precision computing unit 2304, and multimodal flow unit 2401 in multimodal extension module 240 to the scheduling and control module 250.

[0086] The scheduling and control module 250 is also used to calculate dynamic scheduling instructions through the hybrid scheduling unit 2502 based on the received real-time load information, and call the execution instruction stream orchestration unit 2501 to send the dynamic scheduling instructions to the calculation module 230.

[0087] Correspondingly, the computing module 230 is further used to schedule idle expert hardware computing units in the hybrid expert model MOE native computing engine, sparse feedforward network array 2303, low-precision computing unit 2304, or multimodal streaming unit 2401 of multimodal extension module 240 to perform computing tasks through multi-pipeline engine 2301.

[0088] Specifically, the expert hardware computing unit dynamic load scheduling mechanism can realize real-time perception and dynamic elastic allocation of the entire chip's computing power resources through information interaction and instruction closed loop of multimodal extension module 240, computing module 230, and scheduling and control module 250. Multimodal extension module 240 can collect real-time load information of each hardware computing unit participating in feedforward network computing in internal multimodal flow unit 2401, and send the real-time load information to computing module 230 to provide multimodal computing power operation status data for global load aggregation. The computing module 230 can aggregate the computing load data of the entire link. On the one hand, it collects the real-time load information of each expert hardware computing unit participating in the feedforward network calculation in the internal hybrid expert model MOE native computing engine 2302, sparse feedforward network array 2303 and low-precision computing unit 2304. On the other hand, it can receive the real-time load information of the multimodal flow unit 2401 hardware computing unit reported by the multimodal extension module 240. After completing the aggregation of all load data, the computing module 230 sends all real-time load information to the scheduling and control module 250 in a unified manner, providing a complete computing power status basis for global scheduling decisions.

[0089] After receiving the real-time load information reported by the computing module 230, the scheduling and control module 250 performs load status analysis and dynamic scheduling instruction calculation through its internal hybrid scheduling unit 2502. It then calls the execution instruction stream orchestration unit 2501 to generate standardized hardware-executable dynamic scheduling instructions and sends these instructions to the computing module 230, completing the global scheduling decision output based on the real-time computing power status. Upon receiving the dynamic scheduling instructions, the computing module 230 executes scheduling actions through its internal multi-pipeline engine 2301. This coordinates the scheduling of idle expert hardware computing units in the hybrid expert model MOE native computing engine 2302, the sparse feedforward network array 2303, and the low-precision computing unit 2304, as well as idle expert hardware computing units in the multimodal stream unit 2401 of the multimodal extension module 240, to execute pending computing tasks. This achieves on-demand dynamic allocation of computing resources, ensuring efficient execution of inference tasks.

[0090] Furthermore, the scheduling and control module 250 also includes a fault detection unit 2503.

[0091] The fault detection unit 2503 is used to collect the running data of each module in the large model inference acceleration chip 200 in real time and perform fault detection based on the collected running data. When a faulty module is detected, the fault-free replacement mechanism is triggered to switch the current task of the faulty module to the backup module for execution, and fault information is generated and sent to the AFD collaborative interface module 210.

[0092] The AFD collaborative interface module 210 is also used to send the received fault information to the attention cluster to ensure the continuity of large model inference.

[0093] Specifically, the fault tolerance and continuous inference assurance mechanism of the large model inference acceleration chip 200, through a full-process design of real-time fault detection, seamless task switching, and collaborative fault status reporting, ensures uninterrupted inference services when a local module of the chip fails, thereby improving the operational reliability and business continuity of the large model inference acceleration chip 200. The fault detection unit 2503 is the core execution unit for chip fault management, which can collect operational data from each module in the large model inference acceleration chip 200 in real time, realizing real-time perception of the operational status of all chip modules. Based on the collected operational data of each module, fault detection can be performed to identify and determine faulty modules that are experiencing operational anomalies. When a faulty module is detected, on the one hand, a fault-free replacement mechanism is triggered to switch the current task carried by the faulty module to a backup module for continued execution, making the fault transparent to the business layer and not interrupting the inference process. On the other hand, fault information can be generated synchronously and sent to the AFD collaboration interface module 210. The AFD collaboration interface module 210 can then forward the received fault information to the attention cluster, enabling the upstream attention cluster to promptly perceive the fault status and task adjustment of the chip and adjust the scheduling logic of subsequent attention operations accordingly, ensuring the continuity of large model inference from the end-to-end level.

[0094] Furthermore, the large model inference acceleration chip 200 also includes an inter-chip synchronization unit 2504.

[0095] The inter-chip synchronization unit 2504 is used to achieve timing alignment with other large model inference accelerator chips 200 based on a dedicated synchronization signal when the deployed large model inference accelerator chip 200 is interconnected with other large model inference accelerator chips 200.

[0096] Specifically, the inter-chip synchronization unit 2504 belongs to the scheduling and control module 250 of the large model inference acceleration chip 200. It is a dedicated timing management unit that supports multi-chip interconnection and collaboration. The inter-chip synchronization unit 2504 is suitable for scenarios where the currently deployed large model inference acceleration chip 200 is interconnected with other large model inference acceleration chips 200 and jointly undertakes large model inference tasks. In this scenario, the inter-chip synchronization unit uses a dedicated synchronization signal as the interaction carrier to achieve timing alignment with other interconnected large model inference acceleration chips 200. This ensures that the execution timing of multiple interconnected large model inference acceleration chips 200 is consistent when collaboratively executing inference tasks. It can avoid problems such as data interaction anomalies and misaligned calculation results caused by timing deviations, and provide a timing foundation for the stable operation of large model inference under a multi-chip architecture.

[0097] The large-model inference acceleration chip in this invention is designed based on a separate architecture for attention and feedforward networks. It includes an AFD collaborative interface module, an on-chip hierarchical storage module, a scheduling and control module, a computation module, and a multimodal extension module. The AFD collaborative interface module interfaces with the attention cluster, receiving the activation values ​​of the current computation task, the hybrid expert routing table, quantization scaling parameters, feature value sequences, and task description information. These are then distributed to the corresponding storage units of the on-chip hierarchical storage module and the scheduling and control module, respectively. The computation results are then sent back to the attention cluster, achieving bidirectional high-speed interconnection with the attention cluster. The on-chip hierarchical storage module includes primary and secondary storage units and a hardware compression and decompression unit. The primary storage unit uses streaming read / write and write-after-read-now-after-write modes to achieve low-latency access. The hardware compression and decompression unit can perform streaming compression storage and decompression output on long activation value sequences. The scheduling and control module, through a hybrid scheduling unit and a deterministic execution instruction stream orchestration unit, generates matching computation instructions based on the received task description information and sends them to the computation module. The computing module schedules multiple expert hardware computing units through a multi-pipeline engine to complete hybrid expert routing and distribution, feedforward network computation, and low-precision quantization. The multimodal extension module can complete multimodal feature value sequence computation according to the coprocessing instructions issued by the computing module, supporting long context data processing. The chip also supports dynamic computing power scheduling based on the real-time load of each expert hardware computing unit, seamless fault replacement and inference continuity assurance, and multi-chip interconnection timing alignment. In addition, the large model inference system includes at least one of the above-mentioned large model inference acceleration chips and an attention cluster, with the attention cluster connected to the AFD collaborative interface modules of each of the major model inference acceleration chips. This technical solution relies on the native collaborative architecture of attention-feedforward separation, combined with intelligent hierarchical storage and a dedicated hybrid expert computing engine design, to significantly reduce data interaction latency, improve storage and computing power utilization efficiency, achieve high throughput and ultra-low latency large model inference, and support multimodal extension and long context acceleration. With dynamic load scheduling, seamless fault replacement, and multi-chip synchronization capabilities, it has strong scenario adaptability and high operational reliability, effectively reducing the overall cost of large-scale model deployment.

[0098] Example 3

[0099] Figure 3 This is a schematic diagram of the structure of a large model inference system provided in Embodiment 3 of the present invention, as shown below. Figure 3 As shown, it includes: a large model inference acceleration chip 310 and an attention cluster 320.

[0100] Attention cluster 320 is connected to AFD collaborative interface module 3101 in each of the various model inference acceleration chips 310.

[0101] Specifically, the basic architecture of the large-model inference system adopts a clustered collaborative inference architecture, which can include two types of core computing hardware entities: large-model inference acceleration chips 310 and attention clusters 320. These two types of core computing hardware entities respectively undertake different types of computing tasks in the large-model inference process, working together to complete the complete large-model inference computation. The attention clusters 320 are connected to the AFD collaborative interface modules 3101 in each of the large-model inference acceleration chips 310 within the system. That is, the attention clusters 320 can establish independent bidirectional interaction paths with each chip through the AFD collaborative interface module 3101 built into each large-model inference acceleration chip 310, providing a hardware connection foundation for computational data transmission and control signaling interaction. This supports the collaborative flow of attention computation and chip-side computation, ensuring the orderly execution of the entire large-model inference process. Furthermore, multiple large-model inference acceleration chips 310 can be interconnected and networked to form a hardware cluster, completing the coordination of inference tasks and data synchronization among multiple chips.

[0102] The technical solution of this invention sets up independent attention clusters and interfaces them with the AFD collaborative interface modules inside each of the large model inference acceleration chips. This allows the attention calculation tasks, which have extremely high computational overhead in large model inference, to be centrally processed by dedicated attention clusters, without occupying the computational resources of the inference acceleration chip itself. This decouples the main inference computation from the attention computation hardware, reduces the computational load of a single acceleration chip, and enables high-speed and stable data interaction between the chip and the attention clusters through the AFD collaborative interface. This significantly improves the throughput of parallel inference for large models, reduces inference latency caused by single-card computational power squeezing, and effectively improves the overall operating efficiency of the multi-chip distributed large model inference system.

[0103] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A large-model inference acceleration chip, characterized in that, include: Multimodal extension module, scheduling and control module, on-chip hierarchical storage module, computing module, and AFD collaborative interface module for attention and feedforward network separation architecture; This AFD collaboration interface module is used to connect to attention clusters; The scheduling and control module is bidirectionally interconnected with both the computing module and the AFD collaborative interface module. The on-chip hierarchical storage module, computing module, and multimodal expansion module are bidirectionally interconnected in pairs. A unidirectional data connection is established between the AFD collaborative interface module and the on-chip hierarchical storage module, and a unidirectional data connection is established between the computing module and the AFD collaborative interface module. The on-chip hierarchical storage module includes a primary storage unit and a secondary storage unit, wherein: The AFD collaborative interface module is used to receive the activation value, hybrid expert routing table, quantization scaling parameters and feature value sequence of the current computing task calculated in real time by the attention cluster, send them to the matching storage unit in the on-chip hierarchical storage module, and send the current computing task description information sent by the attention cluster to the scheduling and control module. The attention cluster is used to calculate activation values, hybrid expert routing tables, and quantization scaling parameters in real time based on the multimodal raw data of the current computing task from external input, and to generate matching description information of the current computing task. The scheduling and control module is used to generate computing instructions and send them to the computing module based on the received current computing task description information; The computation module is used to respond to computation instructions, and perform hybrid expert route distribution and feedforward network computation for the current computation task based on the activation value, hybrid expert routing table, quantization scaling parameters and feature value sequence read from the on-chip hierarchical storage module. It also issues coprocessing instructions to the multimodal extension module according to the task type of the current computation task, and feeds back the computation results to the AFD cooperative interface module in real time. The multimodal extension module is used to obtain the feature value sequence corresponding to the current computing task from the matching storage unit in the on-chip hierarchical storage module according to the coprocessing instructions issued by the computing module, perform feature value sequence calculation, and return the calculation result to the computing module. The AFD collaborative interface module is also used to send the real-time received calculation results back to the attention cluster for the attention cluster to perform subsequent attention calculations. Among them, the AFD cooperative interface module includes an attention-to-feedforward network interface and a communication direct memory access controller; The attention-to-feedforward network interface is used to receive activation values, hybrid expert routing tables, quantization scaling parameters, and feature value sequences sent by the attention cluster. The communication direct memory access controller is used to distribute the activation values, hybrid expert routing table, quantization scaling parameters and feature value sequence obtained from the attention-to-feedforward network interface to the matching memory cells in the on-chip hierarchical memory module. The on-chip hierarchical storage module also includes hardware compression and decompression units. The first-level storage unit is an L1 streaming static random access memory, and the second-level storage unit is an L2 shared static random access memory, or an L2 shared dynamic random access memory. The first-level storage unit is configured to use a streaming read / write mode and a write-after-read mechanism to achieve low-latency data reading. The communication direct memory access controller is specifically used to write the activation values ​​obtained from the attention to the feedforward network interface into the first-level storage unit in batches according to the activation flow order; and to write the hybrid expert routing table, quantization scaling parameters and various hybrid expert weights used by the computing module obtained from the attention to the feedforward network interface into the second-level storage unit according to the address mapping rules, so that the computing module can read the data through the dedicated address line during subsequent calculations. The hardware compression and decompression unit is specifically used to perform streaming hardware compression processing on the long activation value sequence, obtain simplified compressed data and write it to the secondary storage unit, and decompress the target simplified compressed data before sending the target simplified compressed data stored in the secondary storage unit to the data requester.

2. The large model inference acceleration chip according to claim 1, characterized in that, The AFD collaborative interface module also includes a feedforward network to the attention interface, a ping-pong buffer unit, and an AFD signaling control unit, wherein: The feedforward network to the attention interface is used to send the computation results sent by the computation module back to the attention cluster; The communication direct memory access controller is also used to aggregate the computation results sent in real time by the computing module to the feedforward network to the attention interface; The ping-pong buffer unit is used to buffer the data flowing through the attention-to-feedforward network interface and the feedforward network to the attention interface to avoid interface data congestion; The AFD signaling control unit is used to establish a bidirectional interconnection mechanism between the AFD collaborative interface module and the attention cluster based on a preset communication protocol.

3. The large model inference acceleration chip according to claim 2, characterized in that, The computing module specifically includes: a multi-pipeline engine, a hybrid expert model MOE native computing engine, a sparse feedforward network array, and a low-precision computing unit. Among them, the hybrid expert model MOE native computing engine, the sparse feedforward network array, and the low-precision computing unit all include multiple expert hardware computing units of different types. The expert hardware computing units in the MOE native computing engine of the hybrid expert model are used to adapt to the characteristics of the hybrid expert algorithm and complete the complex comparison and nonlinear operation computing tasks required for the selection of top high-scoring experts. The expert hardware computing units in the sparse feedforward network array are used to perform the conventional numerical calculation task of basic matrix multiplication. The expert hardware computing units in the low-precision computing unit are used to perform low-precision quantization and dequantization tasks. The multi-pipeline engine is used to respond to computation instructions, retrieve the hybrid expert routing table from the secondary storage unit, determine the type and target number of expert hardware computing units to execute the current computation task based on the task characteristics of the current computation task, and select the target execution unit from the hybrid expert model MOE native computing engine, sparse feedforward network array and low-precision computing unit according to the determined type of expert hardware computing unit, and send expert hardware computing unit allocation instructions to the target execution unit. The target execution unit is used to trigger the activation of the target number of expert hardware computing units according to the target number of expert hardware computing units carried in the expert hardware computing unit allocation instruction issued by the multi-pipeline engine, and to obtain the activation value, quantization scaling parameter and feature value sequence from the primary storage unit and the secondary storage unit respectively and distribute them to the activated expert hardware computing units to execute the computing task.

4. The large model inference acceleration chip according to claim 3, characterized in that, The multimodal extension module specifically includes a multimodal streaming unit, a long context acceleration unit, and a shared buffer unit. The multimodal streaming unit internally includes multiple expert hardware computing units, which are used to perform computational tasks corresponding to multimodal raw data of speech or image data types. The long context acceleration unit is used to read the feature value sequence and activation value sequence matching the current computing task from the primary storage unit according to the received coprocessing instructions, and save them to the shared cache unit. During the data saving process, the feature value sequence and activation value sequence that overflow the shared cache unit space read from the primary storage unit are compressed and saved. Before the multimodal stream unit obtains the compressed data from the shared cache unit, the compressed data is decompressed. The multimodal flow unit is used to trigger the activation of a number of matching hardware computing units according to the coprocessing instructions of the current computing task sent by the computing module, and to obtain the matching feature value sequence and activation value sequence from the shared cache unit, distribute them to the activated hardware computing units to perform feedforward network computing, and return the computing results to the computing module.

5. The large model inference acceleration chip according to claim 2, characterized in that, The scheduling and control module specifically includes: a deterministic instruction stream orchestration unit and a hybrid scheduling unit, wherein: The hybrid scheduling unit is used to call the instruction orchestration tool provided by the deterministic execution instruction stream orchestration unit based on the received current computing task description information, generate computing instructions that match the task type of the current computing task, and send them to the computing module.

6. The large model inference acceleration chip according to any one of claims 1-4, characterized in that, The multimodal extension module is used to send the real-time load information of each hardware computing unit participating in the feedforward network computing in the multimodal flow unit to the computing module; The computing module is used to send the real-time load information of each expert hardware computing unit participating in feedforward network computing in the hybrid expert model MOE native computing engine, sparse feedforward network array, low-precision computing unit and multimodal flow unit in multimodal extension module to the scheduling and control module. The scheduling and control module is also used to calculate dynamic scheduling instructions through the hybrid scheduling unit based on the received real-time load information, and call the execution instruction stream orchestration unit to send the dynamic scheduling instructions to the calculation module; Correspondingly, the computing module is further used to schedule idle expert hardware computing units in the MOE native computing engine of the hybrid expert model, the sparse feedforward network array, the low-precision computing unit, or the multimodal streaming unit of the multimodal extension module to perform computing tasks through the multi-pipeline engine.

7. The large model inference acceleration chip according to claim 6, characterized in that, The scheduling and control module also includes a fault detection unit; The fault detection unit is used to collect the running data of each module in the large model inference acceleration chip in real time and perform fault detection based on the collected running data. When a faulty module is detected, the fault-free replacement mechanism is triggered to switch the current task of the faulty module to the backup module for execution and generate fault information to be sent to the AFD collaborative interface module. The AFD collaborative interface module is also used to send the received fault information to the attention cluster to ensure the continuity of large model inference.

8. The large model inference acceleration chip according to claim 7, characterized in that, The large-model inference acceleration chip also includes an inter-chip synchronization unit; The inter-chip synchronization unit is used to achieve timing alignment with other large model inference accelerator chips based on a dedicated synchronization signal when the deployed large model inference accelerator chip is interconnected with other large model inference accelerator chips.

9. A large-scale model reasoning system, characterized in that, include: At least one large-model inference acceleration chip and attention cluster as described in any one of claims 1-8; The attention clusters are connected to the AFD collaborative interface modules in the various model inference acceleration chips.

Images