Large model system based on compute-accelerated chips

Abstract

The application discloses a large model system based on a computing acceleration chip, and relates to the field of large models. A plurality of computing acceleration units and a management server are used to form and implement an inter-chip and off-chip data interaction network; a hybrid display memory of the computing acceleration unit is matched with an SSD to form a multi-source storage mode; the computing chip is internally provided with a normalized on-chip network, an interconnection transmission system, a storage control system, and a plurality of computing acceleration cores; the normalized on-chip network can read model parameters of a target position based on the storage control system and send the model parameters into the computing acceleration cores for calculation and storage; the interconnection transmission system is used to interact with the management server and the remaining computing acceleration units, and read and store external model parameters and inter-chip model parameters. Through the collaborative design of the hybrid display memory architecture and the normalized on-chip network, in combination with a multi-level routing control and a dynamic configuration mechanism, the technical problems of insufficient display memory capacity, excessively high hardware cost and limited transmission bandwidth in the traditional scheme are effectively solved.

Description

Large-scale model system based on computing acceleration chip Technical Field

[0001] This application relates to the field of large modeling, and in particular to a large modeling system based on a computing acceleration chip. Background Technology

[0002] Large models meeting commercial requirements typically require models with hundreds of billions of parameters. Considering the memory bottleneck of large models, parameter storage employs full video memory (early GPUs were used as graphics cards, so all memory directly accessible by the accelerator chip is called video memory), with full HBM storage being the mainstream approach. Due to the limited video memory of a single accelerator device (single card / single GPU), models with hundreds of billions of parameters often require multiple multi-card servers for inference. As shown in Figure 1, a typical 8-card server architecture has high construction costs in server interconnection, and inference systems based on multiple servers are more difficult to develop in terms of software, making the development threshold unfriendly to small and medium-sized enterprises. The introduction of the H200 / H20 141GB video memory version has increased the upper limit of video memory capacity of a single server, enabling full video memory solutions for models such as llama3-405B and deepseek-v3-671B to be implemented on a single server, reducing system costs. However, the integration of high-density HBM places higher demands on the manufacturing process of computing devices / chips.

[0003] Large-scale models, exemplified by Deepseek-v3, utilize the MOE architecture, which significantly reduces operating costs. Its open-source nature not only improves performance but also lowers operating costs, making low-cost all-in-one machines possible. Figure 2 illustrates a typical all-in-one machine structure based on the MOE architecture. In this all-in-one machine, the parameters of the MLA (Multi-Layer Perceptron) are still stored in the GPU's GDDR (consumer-grade graphics cards use GDDR, while data center-grade graphics cards use HBM, which has higher bandwidth but also higher cost). The MLP (Multi-Layer Perceptron) parameter design leverages the reduced bandwidth-to-capacity ratio requirement of the MOE model, using DDR instead of the expensive HBM / GDDR. Taking the 256 expert models of the Deepseek-v3 model as an example, 8 are selected for each calculation using a routing algorithm, reducing the bandwidth-to-capacity ratio requirement to 1 / 16 of the original. However, this all-in-one machine design is a temporary solution. The DDR is mounted on the CPU, limited by the maximum 16 lanes of PCIe communication bandwidth between the CPU and GPU. If the MLP weights in the DDR are transferred to the GPU for computation, the bandwidth-to-capacity ratio will further decrease significantly; if the bandwidth-to-capacity ratio is maintained, the MLP computation needs to be performed within the CPU, resulting in a significant decrease in computing power. Therefore, although the all-in-one architecture design is low-cost, it has the problem of difficulty in expansion, and there is also a certain degree of mismatch in hardware resources.

[0004] Neither of the above two solutions addresses the issue of increased KV cache size in models with hundreds of billions of records. Taking the deepseek-v3 model as an example, the basic model has 61 layers (excluding the MTP layer), with each layer containing 128 heads of key-value pairs. The key dimension is 192, and the value dimension is 128. The data format is BF16, which is 2 bytes. Considering a default maximum sequence length of approximately 160KB, the KV cache space required for a single task is approximately (192 + 128) * 2 bytes * 128 heads * 61 layers * 160KB ≈ 800GB. Compared to the mainstream 80GB of VRAM on a single data center GPU and 24-48GB on a consumer-grade GPU, this is a huge resource consumption even for a single task, and the cost skyrockets under multi-tasking. Failing to use a KV cache would lead to a surge in computational load. On the algorithm side, the Deepseek-v3 model introduces the LoRa algorithm to compress the storage capacity required for the KV cache, balancing storage capacity requirements and computational performance. Specifically, it changes the key-value storage to storing an intermediate compressed KV, shared by 128 heads. During computation, the key and value are multiplied by the corresponding weights in real time to calculate the key and value. After compression, the required storage is 576 * 2 bytes * 61 layers * 160K ≈ 11GB, a reduction of orders of magnitude. However, with the length of the sequence, the key and value of the previous tokens need to be recalculated in real time for each token, resulting in an increasing MLA computation load and a significant performance degradation. Summary of the Invention

[0005] This application provides a large model system based on a computing acceleration chip, which has the advantages of reducing hardware costs, improving storage bandwidth utilization, supporting flexible expansion architecture, and optimizing model parameter transmission efficiency compared to traditional model systems.

[0006] The large model system based on the computing acceleration chip includes P computing acceleration units with direct access memory groups and external SSD storage, and Q management servers; the memory groups are hybrid memory of HBM, GDDR, DDR and LPDDR, and are used with external SSD storage to access model parameters.

[0007] P computing acceleration units are connected to an inter-chip interconnection switching system for inter-chip data exchange; P computing acceleration units and Q management servers are respectively connected to an Ethernet switching system, and the target management server sends instructions and tasks to the target computing acceleration unit through the Ethernet switching system and performs external data exchange; each computing acceleration unit is equipped with a computing acceleration chip, connects to external SSD storage through an SSD interface, connects to the inter-chip interconnection switching system through a network port, and connects to the inter-chip interconnection switching system through an inter-chip interconnection port;

[0008] The computing chip includes a normalized on-chip network, an interconnection and transmission system connected to the normalized on-chip network, a storage control system, and several computing acceleration cores. The normalized on-chip network reads model parameters at the target location based on the storage control system and sends them to the computing acceleration cores for calculation and storage of model data. The normalized on-chip network interacts with the management server and other computing acceleration units through the interconnection and transmission system to read and store external model parameters and inter-chip model parameters.

[0009] The beneficial effects of the technical solution provided in this application include at least the following: the system architecture composed of a management server and multiple computing acceleration units can distribute and accelerate the computation of large-scale model inference tasks in parallel, and the inter-chip interconnection and exchange system can be used to complete sub-tasks and inter-chip data interaction. In the core part of the computing unit, the computing acceleration chip integrates a storage control system, an interconnection and transmission system, and a normalized on-chip network. Through the collaborative design of a hybrid memory architecture and a normalized on-chip network, combined with multi-level routing control and dynamic configuration mechanisms, the technical problems of insufficient memory capacity, high hardware cost, and limited transmission bandwidth in traditional solutions are effectively solved. This approach offers advantages such as reduced hardware cost, improved storage bandwidth utilization, support for flexible expansion architecture, and optimized model parameter transmission efficiency. Attached Figure Description

[0010] Figure 1 shows a typical 8-card server architecture;

[0011] Figure 2 shows a typical all-in-one structure of the MOE architecture;

[0012] Figure 3 is an overall structural diagram of a large model system based on a computing acceleration chip provided in an embodiment of this application;

[0013] Figure 4 shows a schematic diagram of another large-scale computing acceleration chip architecture;

[0014] Figure 5 is a schematic diagram of the configuration for communication with the external environment using the computing acceleration chip as a whole.

[0015] Figure 6 shows a schematic diagram of the storage control system in one possible form;

[0016] Figure 7 shows a schematic diagram of the structure of an interconnected transmission system in one possible form;

[0017] Figure 8 shows a schematic diagram of the normalized on-chip network and its interconnection configuration;

[0018] Figure 9 is a schematic diagram of the network structure for storing routes in this application;

[0019] Figure 10 shows a schematic diagram of an optimized storage routing network structure;

[0020] Figure 11 shows the flowchart of the normalization and denormalization pipeline operations performed by various normalization modules;

[0021] Figure 12 shows a schematic diagram of the structure of a computational routing receiving network;

[0022] Figure 13 shows a schematic diagram of the computational routing transmission network;

[0023] Figure 14 illustrates a schematic diagram of data processing for a normalized on-chip network;

[0024] Figure 15 shows a schematic diagram of the routing management module in one possible form. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0026] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0027] Figure 3 is an overall structural diagram of the large model system based on a computing acceleration chip provided in an embodiment of this application. The system includes P computing acceleration units, Q management servers, an inter-chip interconnection switching system, and an Ethernet switching system. The P computing acceleration units are connected to the inter-chip interconnection switching system, enabling data exchange between any two computing acceleration units or between one and many. The P computing acceleration units and Q management servers are respectively connected to the Ethernet switching system, enabling data communication between any management server and any other management server and computing acceleration unit through the switching network. For tasks involving model inference calculations, the target management server sends instructions and tasks to the target computing acceleration unit through the Ethernet switching system and performs data exchange.

[0028] It should be noted that the management server in this application can be a management server or management CPU with higher scheduling authority located above the accelerator card, used for task allocation and transmission of necessary model parameters. The management CPU relies on the management server and actually undertakes management tasks, but a single management server typically has multiple management CPUs. For scenarios with low load requirements, a single physical CPU can be virtualized into several or even a dozen CPUs as management endpoints. In the above design, model data can be stored in the SSD of each computing unit. Thus, when importing the model, each computing unit can import weights from its local storage to the GPU memory, resulting in a shorter path and parallel execution, leading to lower latency.

[0029] The inter-chip interconnect switching system in this application embodiment can be implemented using an NVLink switching chip + NV Switch network architecture, or other high-speed transmission protocols can be used as supplementary protocols. NVLink technology has become the mainstream choice, especially in ultra-large-scale model training, so this application will not elaborate on it further. The Ethernet switching system can use NICs (Network Interface Cards) to connect to single-layer or multi-layer Ethernet switches, and then the Ethernet switches communicate with the management server. For example, the NVIDIA Spectrum-X platform, through its optimized Ethernet architecture, supports scale-up (interconnection of accelerator cards within a single node) and scale-out (interconnection between multiple nodes), meeting the communication needs of large-scale models. This system architecture is a commonly used network architecture, and this application will not elaborate on it further.

[0030] The computing acceleration unit is centered around a computing accelerator card and connects to several external SSD storage devices via SSD ports, while the memory modules are typically located inside the computing accelerator card. Figure 4 shows a schematic diagram of the computing accelerator card architecture. Each computing accelerator chip is configured with several memory modules according to requirements. The computing accelerator unit is designed with computing accelerator chips, connects to external SSD storage via an SSD interface, and accesses an inter-chip interconnect switching system via a high-speed network port, enabling data communication between the accelerator card and external systems.

[0031] The memory array in this application is composed of a variety of storage devices, including but not limited to a hybrid memory array consisting of HBM, GDDR, DDR and LPDDR, which is used together with external SSD storage to access model parameters (including KV Csche, token, various expert model weight parameters, etc.).

[0032] Figure 5 is a schematic diagram of the overall structure of the computing acceleration chip, which is divided into a computing plane and a data plane. The data plane includes a normalized on-chip network, an interconnect transmission system connected to the normalized on-chip network, and a storage control system. The computing plane integrates several computing acceleration engines / cores for accelerating model calculations and handling high-concurrency calculations of vector, scalar, and matrix data.

[0033] The storage control system is a cluster of controllers that supports the collaborative operation of multiple storage protocols. Based on various built-in storage controllers and their related protocols, it directly accesses memory modules and external SSD storage through corresponding storage controller connections for retrieving model parameters. For example, it can be implemented using a combination of NVMe / memory expansion CXL controllers, PCIe systems, and independent HBM / GDDR / DDR controllers, allocating storage address space through an interleaved mode. This system uses differentiated controller configuration to allocate parameters with the highest bandwidth-to-capacity ratio requirements to HBM memory, while parameters with lower requirements are allocated to DDR / LPDDR or even external SSDs. Compressed KV cache or raw KV cache required for long text is stored on external SSDs to balance bandwidth requirements and storage costs.

[0034] The memory array in this application is a hybrid memory array composed of heterogeneous storage media such as HBM, GDDR, DDR, and LPDDR. Specifically, it can be implemented using physical layer interface compatibility design and logical layer interleaving addressing technology. This combination accesses the model parameters by matching the media characteristics. HBM carries high bandwidth-to-capacity ratio requirement model parameters such as multi-head attention (MLA) calculation, while DDR / LPDDR stores routing expert models and compressed KV caches, reducing the performance bottleneck of a single storage medium.

[0035] Normalized on-chip networking is an intelligent routing architecture used to uniformly manage intra-chip, inter-chip, and extra-chip data transmission paths. The interconnection and transmission system is a crucial module for data communication with external devices. Specifically, it can be designed to support multi-protocol communication through on-chip and off-chip data transmission channels, for example, using PCIe endpoint devices, high-speed Ethernet controllers, and programmable inter-chip interconnection modules, while maintaining compatibility with traditional acceleration devices through a protocol conversion layer. This system overcomes the physical limitation of eight cards per machine by dynamically segmenting and recombining data streams, enabling flexible configuration of the computing topology and supporting distributed access to model parameters within the acceleration chip cluster. This application considers the computing accelerator card as a whole, and its data interaction and accelerated computation can be divided into the following three levels:

[0036] 1. On-chip acceleration: When the computing acceleration card of this application performs acceleration tasks as a whole, the normalized network reads the on-chip model parameters at the target location based on the storage control system and sends them to a specific computing acceleration core for calculation, as well as storing the on-chip model data back.

[0037] 2. Off-chip acceleration: When the computing accelerator card needs to coordinate with an external management server / CPU, the normalized on-chip network interacts with the external management server based on the interconnection transmission system, receives instructions and tasks, and reads and stores external model parameters.

[0038] 3. Inter-chip acceleration: When a computing accelerator card needs to interact with other computing accelerator cards in the system, the normalized on-chip network also interacts with other computing accelerator chips based on the interconnection transmission system to read and store inter-chip model parameters.

[0039] In this embodiment, the model data within, between, and outside the chip can be the original token, the KV cache generated during the calculation process, various model weight parameters, intermediate data generated by multi-task parallel computing, and calculation results, etc. The specific type is determined according to the task and different stages.

[0040] As shown in Figure 5, in some embodiments, the computing acceleration chip can also have a built-in management plane, specifically including a routing management module. This module can dynamically optimize the routing data transmission path of the normalized on-chip network and schedule system bandwidth resources according to the received instruction tasks. The accelerator card first allocates parameters to different storage media based on model characteristics and computing tasks. High-frequency access MLA parameters are stored in HBM, expert model parameters that only access a portion of the data each time are stored in DDR / LPDDR, and large-capacity data is stored in external SSD storage. During computation, the normalized on-chip network reads the corresponding parameters from the hybrid memory according to task requirements, preprocesses them, and then sends them to the computing acceleration core. For parameters that need to be accessed across chips, efficient transmission is achieved through the interconnect transmission system. The computation results are written back to the corresponding storage location via storage routing. The entire process is dynamically optimized by the intelligent routing management module to ensure maximum data transmission efficiency.

[0041] In the above technical solutions, the use of hybrid memory significantly increases the upper limit of the accelerator card's memory, enabling the computation of models with tens of billions of elements with less hardware, and fully leveraging the system cost-effectiveness improvement brought by MOE computing. The use of multi-parallel SSD direct access expands the available storage space of the KV cache in multi-computing chip scenarios, improving performance while reducing the number of computing chips required and lowering the system TCO. Simultaneously, while maintaining low cost in a minimal system (single accelerator chip), the use of parallel SSD direct access to expand DDR / LPDDR memory space accelerates the full cache model weights and KV cache within the card, rather than utilizing the memory managing the CPU, thus making more rational use of hardware resources and providing higher computing performance.

[0042] In the above structure, the collaborative management of hybrid video memory and external SSD storage suffers from interface protocol differences, access mode conflicts, and insufficient bandwidth utilization, resulting in low model parameter access efficiency and limited flexibility. As shown in Figure 6, in a possible implementation, the storage control system can be configured in detail as follows:

[0043] The storage control system is configured to include several NVMe / memory extension CXL DDR controllers and corresponding PCIe RC subsystems, several DDR / LPDDR / GDDR controllers, and several HBM controllers. The NVMe controllers are specifically used to control SSD storage. Specifically, when the PCIe RC subsystem enters RC master mode, it reads or writes back model data from the external SSD storage via the NVMe controllers. The DDR / LPDDR / GDDR controllers read or write back model data from the DDR / LPDDR / GDDR storage according to instructions; the HBM controllers read or write back model data from the HBM storage according to instructions.

[0044] Specifically, in the above configuration, the combination of the NVMe / memory extension CXL DDR controller and the PCIe RC subsystem enables the chip to actively initiate SSD storage access in RC master mode. The acceleration capabilities of the NVMe controller, compared to traditional SSD read speeds, are improved in the following ways:

[0045] 1. The task queue is located inside the controller, rather than in memory, resulting in shorter read / write paths and lower latency;

[0046] 2. The data cache area is also located inside the controller, resulting in a shorter access path and lower latency;

[0047] 3. Since the real-time processing capability of normalized data requires less data buffer space, it is possible to integrate the data buffer inside the controller;

[0048] 4. SSD addresses are mapped to the storage network through a unified addressing system using logical addresses. Compared to the traditional approach where logical addresses are stored as a form in the file system, accessing SSD data involves first reading and writing the file system, and then reading and writing the actual data, resulting in fewer steps and faster access.

[0049] Access to DDR / LPDDR / GDDR video memory does not rely on the PCIe system; it is an on-chip data transfer that can be scheduled directly according to demand. This application does not impose any restrictions or elaborate on the specifics.

[0050] Traditional PCIe-based architectures are limited by a minimum system size of eight cards per machine, resulting in the inability to dynamically allocate hardware resources according to task requirements. Furthermore, traditional protocols struggle to balance compatibility, flexibility, and transmission efficiency, leading to poor system scalability, high deployment costs, and low cross-device data transmission efficiency. To address this, this application designs an interconnect transmission system to achieve flexible expansion and efficient transmission. Figure 7 shows a schematic diagram of the interconnect transmission system, which includes a PCIe EP subsystem, a high-speed Ethernet subsystem, an inter-chip interconnect transmission controller, and several multi-access high-speed interconnect channels (e.g., N channels or other numbers, depending on actual requirements).

[0051] The PCIe EP subsystem is designed to be compatible with traditional GPU / accelerator card design patterns, enabling accelerator chips to communicate with the management CPU via PCIe. The high-speed Ethernet subsystem is designed to break the limitation of a minimum single-machine eight-card limit when building computing systems. It allows for the construction of a more flexible CPU + compute chip topology through the network, enabling the rational configuration of the CPU to compute chip ratio based on the computing scenario, and dynamically allocating this ratio as task requirements change, maximizing hardware utilization and reducing deployment costs.

[0052] The PCIe EP subsystem corresponds to the PCIe RC subsystem. When the chip enters EP slave mode, it communicates with the target number of management servers through the high-speed Ethernet subsystem. For example, it uses the PCIe 5.0 protocol to achieve a transmission rate of 64GT / s. This subsystem allows accelerator cards to be seamlessly connected to traditional computing servers by retaining the physical interface and protocol stack of traditional GPUs / accelerator cards.

[0053] The high-speed interconnect channel and its corresponding inter-chip interconnect transmission controller support high-speed inter-chip interconnection. It can be implemented using mature protocols such as NVLink, InfiniBand, and RDMA, or a custom optimized transmission protocol can be used. The high-speed interconnect channel subsystem enables communication up to the transport layer. The inter-chip interconnect transmission controller is responsible for application layer transmission protocols, as well as the segmentation and reassembly of data for multi-channel transmission. For example, the inter-chip interconnect transmission controller incorporates a hardware-level data fragmentation engine, which divides a single model parameter stream into multiple data blocks at a 512KB granularity and distributes them to different high-speed interconnect channels using a round-robin algorithm.

[0054] In traditional architectures, the lack of a unified path management mechanism between different types of storage controllers and external transmission subsystems leads to low storage bandwidth utilization, redundant and inefficient on-chip and off-chip data transmission paths, limited resource allocation flexibility when multiple controllers work together, and difficulty in achieving efficient access and dynamic scheduling of model parameters. To address this, this application introduces a normalized on-chip network and makes targeted improvements to it.

[0055] Figure 8 shows a schematic diagram of the normalized on-chip network and its connection configuration. The normalized on-chip network includes storage routing, computing routing, M on-chip normalization modules connected to the storage control system, N-2 inter-chip normalization modules connected to the interconnection transmission system, and at least 2 off-chip normalization modules.

[0056] For ease of description, the internal and external components are divided into two groups, A and B. Group A contains M on-chip normalization modules, corresponding to a maximum of M storage controllers. Group B contains N modules, of which at least one corresponds to a PCIe EP subsystem, at least one corresponds to a high-speed Ethernet subsystem, and the remaining at most N-2 correspond to inter-chip interconnect transmission controllers.

[0057] The number of on-chip normalization modules corresponds to the maximum number of storage controllers. For storage controllers of the same type, a single on-chip normalization module can connect to multiple controllers, and storage bandwidth optimization is achieved through interleaving mode. For example, when two HBM controllers share the same module, the address space is divided into alternating access regions, and the controllers can access different data blocks in parallel, enabling the total bandwidth to reach the theoretical peak of a single module. Off-chip normalization modules independently interface with the PCIe EP subsystem and the high-speed Ethernet subsystem. The former is compatible with the data encapsulation format of traditional PCIe devices, while the latter supports frame reassembly and verification of the Ethernet protocol stack. Inter-chip normalization modules are directly connected to the interconnect transmission controller, dividing model parameters into fixed-size transmission units according to predefined data segmentation rules, such as dividing the weight matrix into 512-byte data packets by row.

[0058] Since traditional on-chip networks are not compatible with the multi-source heterogeneous storage control strategy used in this application, the interaction between the multi-source heterogeneous storage controller and inter-chip or off-chip modules lacks an efficient pipelined scheduling mechanism, making it difficult to achieve efficient vertical deep transmission and dynamic reassembly of model parameters. Therefore, this application splits the on-chip network into two main parts: storage routing and computation routing.

[0059] The storage route is connected to M on-chip normalization modules, at least two off-chip normalization modules, and the compute route, respectively, for storing and retrieving on-chip model data, off-chip model data, and sending and receiving various model data to and from the compute route. When storing on-chip model data, the storage route assigns priorities based on controller type; for example, HBM controller requests are processed before DDR controller requests. A pipeline mechanism is also used to interleave access requests from multiple controllers. When storing off-chip data, the storage route selects the corresponding off-chip normalization module based on the protocol type; for example, Ethernet data streams enter a buffer queue after passing through a dedicated CRC check unit. When sending data to the compute route, the storage route adds a uniform header identifier to data packets from different sources; for example, it uses 3 bits to identify the data source type and 2 bits to identify the target compute core number.

[0060] The compute routing is connected to N-2 inter-chip normalization modules and compute acceleration cores, which are used to send and receive inter-chip model data, send various received model data to the compute acceleration cores, and obtain output data. Specifically, when processing intra-chip model data access, the storage routing receives access requests from the storage controllers through the intra-chip normalization modules. For example, requests from four HBM controllers are merged into two intra-chip normalization modules, each using a dual-channel interleaving mode, increasing the effective bandwidth to 1.8 times that of a single module.

[0061] In traditional on-chip routing architectures, hierarchical design can lead to complex data transmission paths, insufficient bandwidth utilization, and a lack of efficient pipelined scheduling mechanisms for interactions between multi-source heterogeneous memory controllers and inter-chip or off-chip modules, making it difficult to achieve efficient vertical transmission and dynamic reassembly of model parameters. Therefore, this application provides an improved memory routing architecture.

[0062] Figure 9 is a schematic diagram of the network structure of the storage routing in this application. The storage routing includes at least M+2 levels of vertically distributed node pipelines. Each level of the node pipeline contains M+2 routing nodes that are cascaded end-to-end to form a ring structure. Each level of routing node is connected to an uplink node and a downlink node. The corresponding routing node receives model data transmitted by its respective downlink node and sends out model data through the uplink node.

[0063] For storage routing, considering the matching of data source exchange capabilities between storage routing normalization modules, a maximum of M+2 downlink pipeline rings are designed (more than M+2 are also possible, but the system cannot reach full-load bandwidth). However, in real-world scenarios, it is difficult to reach the design limit, and considering the limited area resources, the number can be appropriately reduced. The ring-shaped transmission method of the routing node pipeline can support data sharing bandwidth replication and distribution to any other normalization module, saving data exchange resources; multi-ring parallelism prevents congestion caused by high-volume transmissions to other transmissions, especially friendly to certain transmissions with high latency performance.

[0064] The top-level node pipeline connects to N-2 inter-chip normalization modules and at least 2 external-chip normalization modules through uplink and downlink nodes to send and receive inter-chip model data and external model data. The bottom-level node pipeline connects to the uplink and downlink nodes to calculate routes and send or receive model data.

[0065] In the M+2 level node pipeline loop, the downlink nodes of the same level routing nodes are cascaded vertically in depth to send inter-chip model data and external model data to the computation route; the uplink nodes of the same level routing nodes are cascaded vertically in reverse in depth to send the model data output from the computation route to the inter-chip and external normalization modules.

[0066] The top-level pipelined ring receives model data from the off-chip normalization module via downlink nodes and then transmits the data to the next level of the pipelined ring. The routing nodes at the same level of each pipelined ring are connected sequentially according to their vertical depth, forming a unidirectional transmission link from the top to the bottom. When data reaches the bottom-level pipelined ring, it is processed by inputting the calculated route through the downlink nodes. The calculation results are transmitted back level by level in the reverse pipeline direction, starting from the uplink nodes of the bottom-level pipelined ring, and finally distributed to the inter-chip or off-chip normalization modules through the top-level pipelined ring. The vertical pipelined cascading structure allows multi-source data to be split into fixed-size data blocks during transmission; for example, each data block can be set to 128 bytes to match the burst transmission length of the HBM controller. The redundant design of the ring routing nodes allows data transmission continuity to be maintained through adjacent nodes in the event of a single node failure. The bandwidth differences between different storage controllers are adapted by dynamically adjusting the pipelined cascading depth; for example, for a GDDR controller with lower bandwidth, the number of vertical pipelined stages can be increased to extend the data transmission time window.

[0067] In some other embodiments, since the M+2 node pipeline rings fully support data transmission modes under various conditions, configuring a separate uplink and downlink node for each routing node would be wasteful. Therefore, Figure 10 shows a schematic diagram of an optimized storage routing network structure. The storage routing also includes at least M+2 levels of vertically distributed node pipeline rings. Each level of the node pipeline ring contains M+2 routing nodes that are cascaded end-to-end to form a ring structure, forming a node matrix.

[0068] The top-level routing nodes in the node matrix connect N-2 inter-chip normalization modules and at least 2 external-chip normalization modules, transmitting and receiving inter-chip model data and external-chip model data. The bottom-level routing nodes connect to the computation route. Each routing node in the node matrix has an uplink port and a downlink port. The downlink ports of routing nodes in the same column are cascaded vertically in depth to send inter-chip and external-chip model data to the computation route. The uplink ports of routing nodes in the same column are cascaded vertically in reverse depth to send the model data output from the computation route to the inter-chip and external-chip normalization modules.

[0069] In the above structure, the uplink and downlink nodes of each routing node are eliminated, and the task of data transmission and forwarding is transferred to the routing nodes. Compared with the previous scheme, the original routing node was designed with 4 ports, while this scheme adds 2 more, using 6 ports to realize the complete transmission scheme, saving system scale and cost investment, while the overall performance is not affected by the support of sufficient pipelined number.

[0070] The aforementioned structure includes a normalization module within the normalized on-chip network. However, the model data from different sources suffers from format differences, inconsistent compression states, and insufficient dynamic adaptation of storage and computing resources. This results in low data transmission efficiency, increased processing latency in the computation acceleration core, and difficulty in effectively alleviating the storage pressure on data transmitted back between and outside the chip. Therefore, Figure 11 illustrates that various normalization modules need to include both normalization pipeline operations and inverse normalization pipeline operations.

[0071] When the chip performs normalized pipeline operation, it parses the received model data packets, determines the target computing acceleration core, synchronizes the parsed data based on the target computing acceleration core, decompresses the data according to the data status, converts the decompressed data according to the target format, reassembles the converted data according to the cutting dimension, and then packages the reassembled data and sends it to the downlink node.

[0072] When the chip performs the inverse normalization pipeline operation, it parses the received model data packets, determines the target computing acceleration chip or target management server, pipelines the data, converts the pipelined data according to the target format, compresses the converted data according to the instructions, segments the compressed data according to the storage task and type, and then packages the segmented data and sends it to the uplink node.

[0073] The six sequential execution steps of normalized pipelined operations can be summarized as follows:

[0074] The first step, when parsing the model data packet, is to determine the physical address of the target computing acceleration core by extracting the packet header identifier;

[0075] The second step, data synchronization, is to adapt to multiple storage controllers. If there is a one-to-one correspondence between the storage controller and the normalization module, then this step is not necessary.

[0076] The third step is to decompress the data and select the corresponding decoder based on the data status identifier. For example, when the data packet is marked as LZ4 compression format, a dedicated hardware decoding unit is called.

[0077] The fourth step, format conversion, converts the data to a precision type supported by the target core, such as dequantizing INT8 quantized data into FP16 format;

[0078] The fifth step is to dynamically adjust the matrix block size based on the cutting dimension parameter, for example, splitting the input tensor into a 128×128 block structure to adapt to the parallel processing unit of the computing kernel.

[0079] The sixth step is to repackage the data and send it to the downstream node.

[0080] The inverse normalization pipelining operation consists of six reverse processing steps. Pipelining converts the calculation results into a continuous data stream, such as arranging scattered matrix blocks in row-major order. The compression step selects a compression algorithm based on the characteristics of the storage medium, such as using a lossless compression algorithm for SSD storage. Data splitting dynamically allocates block sizes based on the storage task type, such as splitting the return data into 4KB aligned blocks to adapt to the block storage characteristics of SSDs.

[0081] In summary, whether it's normalization or inverse normalization, the goal is to convert discrete storage addresses into unified access addresses; to switch data packet formats, such as block storage requiring multiple packets of exactly 4KB; to package / reassemble multiple blocks of data; to convert data format and command format, such as converting read / write commands for storage addresses into read / write commands for NVMe logical blocks; and to adapt the computing core architecture, reassembling data to the design of the computing core's internal pulsating array, sparse matrix computation, etc., so that data can be continuously input and output to the computing core.

[0082] If compression is performed in the above process, the data needs to be compressed / decompressed in the pipeline; if the data is not compressed, the compression / decompression steps can be skipped in the pipeline. For multidimensional reassembly of data packets, in addition to the data structure, there may be data padding. Data padding takes into account the characteristics of matrix calculations, and generally fills in numbers with values ​​of 0, rather than directly filling all bits with 0.

[0083] For the structure of computational routing, since it involves data transmission from multiple heterogeneous storage sources, the design process of a single routing network is too complicated for complex scenarios with high concurrency and high bandwidth. Therefore, this application chooses to further split computational routing into a computational routing receiving network and a computational routing sending network.

[0084] Figure 12 illustrates the structure of the computation routing receiving network. Considering the influencing factors, the traffic demand from GPU memory to the compute core is the largest, accounting for the highest proportion of transmission; the amount of data returned from computation to GPU memory or external management CPU is relatively small; and the data traffic from GPU memory to inter-chip communication is much smaller than the traffic from storage to the compute core. In terms of bandwidth, the theoretical bandwidth of GPU memory is much greater than that of inter-chip communication. Therefore, it is assumed that each port from the storage route can directly reach each compute core / engine through a parallel pipeline. The data streams arriving at the normalization module are merged into one path and then implemented through a one-to-many routing mechanism.

[0085] Specifically, the computational routing receiving network configuration includes at least M+2 receiving node pipelines, S+1 data stream merge nodes, and one distribution routing node. The receiving node pipelines are configured as a linear structure rather than a ring structure, with each receiving node pipeline containing S+1 cascaded receiving nodes.

[0086] M+2 receiving nodes pipelinedly receive various model data output from the storage routing, and transmit the model data pipelinedly through S+1 cascaded receiving nodes. In the first S levels of these M+2 receiving node pipelines, peer-to-peer receiving nodes are connected to the first S level data stream merge nodes. The first S level data stream merge nodes then send the model data into the computation routing transmission network, which ultimately routes and forwards it to the receiving ports of all computation acceleration cores.

[0087] The S+1 level receiving nodes of these M+2 receiving nodes are all connected to the S+1 level data stream merge node. The S+1 level data stream merge node distributes the model data to the computational routing transmission network through the distribution routing node, and finally the computational routing transmission network routes and forwards it to the receiving ports of N-2 inter-chip normalization modules.

[0088] The number of receiving nodes depends on the number of receiving ports of all computing acceleration cores. For example, if each computing acceleration core has k input / output ports, and a total of S input / output ports, then S+1 receiving nodes are needed. The last receiving node is dedicated to a small amount of inter-chip data transmission. Each merge node can aggregate model data from up to M+2 different sources, achieving high concurrency and high redundancy.

[0089] In the initial S-level processing stage, each transmission unit simultaneously outputs the processed data to the same-level data stream merge node. For example, when S=4, the outputs of the first four transmission units enter four independent merge nodes. The data stream merge node merges data from the same stage of multiple receiving node pipelines. For example, it merges the second-level outputs of eight pipelines into a complete data block and sends it to the computation acceleration core. The remaining data after the initial S-level processing enters the (S+1)-level transmission unit, and the final outputs of all receiving node pipelines are converged to the (S+1)-level data stream merge node. For example, when M+2=8, the final outputs of eight pipelines are merged at this node. The distribution routing node dynamically allocates the integrated data to N-2 target modules based on the load status of the inter-chip normalization modules. For example, it selects 14 currently idle modules from 16 inter-chip modules for distribution. This architecture reduces data latency through pipelined transmission, integrates data using multi-level merge nodes, and enhances task adaptability by combining a dynamic distribution mechanism. This enables the computing acceleration core to acquire multiple processed data simultaneously while ensuring cross-chip data transmission efficiency.

[0090] The receiving node pipeline can be implemented using a FIFO structure, the data merge node can be designed using a multiplexer, and the distribution routing node can distribute data based on a lookup table. This design enables the entire receiving network to efficiently process model data from different sources and dynamically adjust the data flow direction according to demand.

[0091] Although the computational routing receives model data distributed by the network to the computational acceleration core and the inter-chip normalization module, there are still problems in how to efficiently integrate the data output by the computational acceleration core and the inter-chip interaction data and transmit them to the target module during the model data transmission process, while ensuring the utilization of transmission bandwidth and the flexibility of dynamic task allocation. These problems include a simple routing structure, limited transmission efficiency, and insufficient multi-path data integration capabilities.

[0092] This application further presents a schematic diagram of the computation routing transmission network shown in Figure 13. For the computation routing transmission network, the traffic between the external interconnect and the computation acceleration core is relatively large, while the traffic returned to storage by the computation acceleration core is relatively small. Therefore, a high-speed interconnect network is designed between computation cores and between the computation core and the external interconnect. Considering that multiple external computation acceleration chips (cards) form a ring network, efficient bypass forwarding and copy forwarding are implemented internally. A high-speed interconnect network is also designed between the external interconnect ports, and the data returning to storage routing uses serial transmission.

[0093] Specifically, the computational routing configuration includes a computational routing transmission network comprising at most S+N-2 levels of vertically distributed node pipelines, one aggregation routing node, and one distribution routing node. Each level of the node pipeline contains (taking two off-chip normalization modules as an example) S+N-2 routing nodes cascaded end-to-end to form a ring structure. Each level of routing node is connected to an uplink node and a downlink node. The corresponding routing node receives model data transmitted by its respective downlink node and sends out model data through the uplink node.

[0094] The top-level node pipeline ring connects to N-2 inter-chip normalization modules and S ports connected to the computing acceleration cores through uplink and downlink nodes to send and receive various model data. The bottom-level node pipeline ring connects to the computing route through uplink nodes and to the aggregation route node through downlink nodes.

[0095] In the S+N-2 level node pipeline loop, the downlink nodes of the same level routing nodes are cascaded vertically in depth to send various model data to the aggregation routing node; the uplink nodes of the same level routing nodes are cascaded vertically in reverse in depth to send the model data output by the computing routing network to the inter-chip normalization module and the computing acceleration core.

[0096] The aforementioned routing nodes are all designed for one-to-many or many-to-one transmission. For one-to-many transmission, each output channel uses one bit as a flag; output is performed when valid, thus supporting duplicate output. For many-to-one mode, a round-robin approach is used, requiring each input port to have a certain data buffering capacity, while also setting polling priorities and ratios to achieve QoS management capabilities. For example, each routing node's downlink can be configured with 8 parallel channels, supporting the transmission of 128 bits of data per cycle. The uplink achieves data distribution through reverse pipeline cascading, for example, by dynamically allocating transmission paths using a priority round-robin mechanism. Aggregation routing nodes can integrate multiple input buffer queues, each with a queue depth of 64 data packets, and schedule data integration using a weighted round-robin algorithm. Distribution routing nodes can be configured to support dynamic port mapping tables, for example, adjusting the correspondence between ports and target modules in real time according to task requirements.

[0097] The computational routing transmit and receive network designed in this application achieves efficient distribution of multi-source heterogeneous model data in computational routing. For the receive network, firstly, the multi-stage pipelined design of the receive nodes improves the parallel reception capability of model data and reduces transmission latency. Secondly, the introduction of multi-stage data flow merge nodes ensures that the computational acceleration core can simultaneously acquire multiple input data, avoiding resource idleness. Finally, through the cooperation of the last-stage merge node and the distribution routing node, dynamic allocation of data between chips is realized, enhancing the system's adaptability to different task requirements. This hierarchical and cascaded receive network structure combined with the dynamic data flow integration mechanism effectively improves the data processing efficiency and flexibility of computational routing.

[0098] For the transmission network, the multi-level ring pipeline structure expands the parallel transmission paths, avoiding the bandwidth limitations of a single channel; the vertical cascading mechanism optimizes the flexibility of timing synchronization and path selection, improving transmission efficiency; and the dynamic allocation capability of aggregation and distribution nodes adapts to changing task requirements, enhancing the ability to reassemble multi-source data. As a result, the efficiency of model data integration and transmission is significantly improved, while ensuring the scalability and real-time performance of dynamic task allocation.

[0099] In some embodiments, similar to Figure 10, the computational route sending network can be further optimized. Specifically, it does not use downlink and uplink nodes, that is, it includes at most S+N-2 levels of vertically distributed node pipeline rings, one aggregation route node and one distribution route node. Each level of the node pipeline ring contains S+N-2 routing nodes that are cascaded end to end to form a ring structure, forming a node matrix.

[0100] The top-level routing nodes in the node matrix connect to N-2 inter-chip normalization modules and S ports connected to the computing acceleration cores, transmitting and receiving various model data. The bottom-level routing nodes connect to the computing routing receiving network and the aggregation routing node, respectively. The routing nodes in the node matrix are equipped with uplink and downlink ports. The downlink ports of routing nodes in the same column are cascaded vertically in depth to send various model data to the aggregation routing node. The uplink ports of routing nodes in the same column are cascaded vertically in reverse depth to send the model data output by the computing routing receiving network to the inter-chip normalization modules and the computing acceleration cores.

[0101] In the above design scheme, all data forwarding and transmission between pipeline rings are handled by the routing nodes. Although this sacrifices some routing functions of the routing nodes, it provides high redundancy and parallel forwarding when there are enough pipeline rings.

[0102] Based on the above design of the storage routing and computation routing network, this application can summarize the following situations according to the data type and execution content, as shown in the data processing diagram in Figure 14.

[0103] Step ①: The on-chip model data is read from the Menory memory group and the external SSD storage by the on-chip normalization module and sent to the storage router.

[0104] Step 2: The stored route sends the model data obtained from each source to the route calculation receiving network;

[0105] Step 3: The routing calculation sending network sends the received model data to the routing calculation sending network;

[0106] Step 4: The routing calculation sending network sends the received model data into the corresponding computing acceleration core to perform relevant accelerated calculations;

[0107] Step 5: The routing calculation sends the results and related model data output by the computation acceleration core to the network.

[0108] Step 6: The routing calculation sending network sends the received relevant model data (including generated temporary data and calculation results, etc.) back to the storage route;

[0109] Step 7: The storage route will receive the relevant model data from the network output of the route calculation and store it back to the target storage location (which can be external SSD storage and Menory memory group).

[0110] Step 8: The storage route receives instructions and tasks from the management server or management CPU, as well as task-related off-chip model data, through the off-chip normalization module.

[0111] Step 9: The stored route sends / feeds the relevant model data of the route calculation to the network output to the management server or management CPU through the off-chip normalization module;

[0112] Step 10: The routing calculation sending network receives inter-chip model data sent by other computing acceleration chips through the inter-chip normalization module;

[0113] Step 11: The routing calculation sending network sends the received relevant model data to other computing acceleration chips through the inter-chip normalization module.

[0114] In all the steps described above, the execution is performed according to the specific functions and tasks. For example, the steps of reading model parameters from external SSD storage, calculating them, and returning them to the management CPU should be steps ①+②+③+④+⑤+⑥+⑨. When it is necessary to receive instructions from the management CPU to transfer model data from DDR to other computing acceleration chips, the steps should be steps ①+②+③+⑪. When it is necessary to obtain other computing acceleration chips to perform accelerated calculations and store them in the SSD, the steps should be steps ⑩+④+⑤+⑥+⑦. The steps for chips to perform other different tasks are not described in detail in this embodiment.

[0115] In some embodiments, the computing acceleration chip architecture accesses and transmits model parameters through a normalized on-chip network, interconnect transmission system, and storage control system. However, during execution, there are problems such as rigid hardware resource allocation, the inability of the transmission network to dynamically adapt to different tasks, and the lack of a model optimization decision mechanism based on data type and usage frequency. This results in insufficient resource utilization, limited task processing efficiency, and an inability to flexibly adjust computing strategies according to actual needs. Referring to Figure 4, this application can also add a model computing management module and a data scoring module to the management plane of the computing acceleration chip.

[0116] Its model computation management module is configured to receive the overall policy set by an external management server, translate the policy into hardware configuration instructions, and dynamically allocate computing resources, storage resources, and bandwidth resources based on model characteristics and system hardware resource status. For example, in a hybrid memory architecture, this module can prioritize the allocation of frequently accessed expert model weights to high-bandwidth storage areas based on the bandwidth difference between HBM and DDR.

[0117] The routing management module is configured to dynamically modify the transmission configuration of nodes within the normalized on-chip network. By adjusting the cascading method of node pipelines or transmission protocol parameters, the transmission network can be adapted to the data transmission modes of different tasks. For example, when performing MOE model inference, the number of transmission channels of the inter-chip normalization module can be temporarily increased to improve the efficiency of expert weight distribution.

[0118] The data scoring module is configured to accelerate the reading frequency of the weight matrix through statistical calculations. It can capture frequently accessed expert model types in real time through a conditional statistical thread within the chip, generating statistical reports containing the weights of the top-k route experts. For example, when the number of expert model calls for a certain sparse activation mode exceeds a preset threshold, an optimization decision is triggered to migrate the model weights to a low-latency storage area.

[0119] In one possible implementation, the routing management module can be configured as shown in Figure 15, specifically including a read information probe unit, an information transmission pipeline, several conditional statistical threads, and a statistical analysis unit. The read information probe unit detects the reading of the weight matrix by the computation acceleration core in real time, and each set of information is transmitted to each conditional statistical thread through the information transmission pipeline. Different conditional statistical threads collect different information according to the configured conditions, and count the number of times the computation acceleration core reads the target route expert weights within a preset time.

[0120] The statistical analysis unit converts statistical information into specific reports based on computing resources. These reports include the topk of high-frequency reading routing expert weights for a single task, the topk of high-frequency reading routing expert weights for several tasks, and the percentage of topk readings in the total. The converted reports are used for decision optimization calculations, and to activate or replace expert models and expert weights.

[0121] Specifically, the read information probe unit can be configured to embed a hardware probe in the physical address access path of the weight matrix. When the computing acceleration core initiates a weight read operation, the probe captures the target weight identifier, access timestamp, and associated task identifier. The information transmission pipeline can adopt a circular buffer structure, with each buffer unit storing triples or quintuples of information for a single weight read event. The buffer depth is set to 1.5 times the maximum theoretical access rate of the computing acceleration core to ensure no data loss occurs under high load scenarios. The number of conditional statistical threads can be set to an integer multiple of the chip's computing acceleration cores. Each thread is bound to an independent memory workspace. Statistical conditions include, but are not limited to, task type, time window, and weight version number. The preset time window can be configured to range from 10 milliseconds to 1 second. The statistical analysis unit incorporates a sliding time window algorithm to aggregate the multi-threaded statistical results over time. The k value of topk is dynamically adjusted based on the currently available computing resources, ranging from an integer between 5 and 20.

[0122] When the computation acceleration core performs inference tasks, the read information probe unit records weight access events in real time through a hardware-level signal capture mechanism, generating a raw data stream containing task ID, weight ID, and timestamp. The information transmission pipeline employs a double-buffering mechanism to alternately receive probe data and distribute it to the memory workspaces of each statistical thread. The conditional statistical thread maps weight IDs to different statistical dimensions according to preset hash rules. For example, when classifying expert weights by task type, a Bloom filter is used to quickly filter target data and count the access frequency within the statistical window. The statistical analysis unit performs global aggregation every 500 milliseconds, normalizing the top-k weights across tasks, calculating their proportion of total access, and generating a visual report containing a weight heatmap and time distribution curve. This report is input into the dynamic scheduling algorithm. When the access ratio of a certain expert weight exceeds 30% for three consecutive statistical periods, a weight preloading mechanism is triggered, migrating high-frequency weights to HBM memory; when the call frequency of a certain expert model is found to be below a threshold, it is removed from the computational resource pool. Through a multi-stage pipeline and parallel statistical architecture, the extraction latency of weighted access features is controlled within 5 milliseconds, which is more than 8 times more efficient than the traditional polling statistical method and 3 times faster in dynamic resource configuration.

[0123] When applying the large model system of this application to a real-world multi-storage hybrid GPU memory scenario, the computation strategy first considers the storage of weight data. Taking the 671B model in deepseek-v3 as an example, the computation of each token only activates approximately 37B of weights. Among these, the MLA weights and the shared expert weights in the MLP are activated for each token computation; while the routing expert weights total 600B+, based on the MOE routing algorithm, only 1 / 32 of each token computation is activated. Based on these characteristics, in the demonstration system design, for a configured sub-computing system (accelerator card architecture), the dynamic weight and cache storage strategy for different scenarios can be designed as follows:

[0124] 1. The number of computing cards is large, and the HBM / GDDR memory capacity reaches the TB level;

[0125] Weight storage: All weights are stored in HBM / GDDR video memory; Kv cache storage: The compressed Kv cache is stored in DDR / LPDDR video memory.

[0126] 2. The number of computing cards is moderate, the HBM / GDDR memory capacity is insufficient, but the DDR / LPDDR memory capacity is in the TB range;

[0127] MLA weights are stored in HBM / GDDR video memory; MLP weights are stored in DDR / LPDDR (if conditions permit, shared expert weights are preferentially stored in HBM / GDDR video memory, and routing expert weights are stored in HBM / GDDR video memory as much as possible); Kvcache is preferentially stored in DDR / LPDDR, and if insufficient, it is stored in SSD.

[0128] 3. The number of computing cards is small, or even only one, and the capacity of HBM / GDDR video memory and DDR / LPDDR video memory is insufficient;

[0129] MLA weights are stored in HBM / GDDR memory; shared expert weights are preferentially stored in HBM / GDDR memory, and secondarily in DDR / LPDDR memory; routing expert weights are stored in DDR / LPDDR memory as much as possible, and secondarily in SSD.

[0130] The computation process of the subsystem corresponding to the accelerator card architecture is as follows:

[0131] 1. The management server / CPU initializes the model according to the resource configuration, manages the import of model weights into the corresponding storage space, and imports model information and calculation strategies into the model calculation acceleration management module inside the chip;

[0132] 2. The management server / CPU determines the token to be computed based on the task, converts it into an embedding, and organizes it into parallel input data (prefilling multiple tokens in a single dialogue, generating a single token in parallel across multiple dialogues, etc.), and downloads the input data to each computing acceleration chip.

[0133] 3. The computing acceleration management module of the computing acceleration chip determines how to perform sub-tasks for the computing of this batch of tokens based on the configuration (a single-layer decoder can be a sub-task, each MLA / MLP can be a sub-task, and there are multiple ways to divide the tasks), and generates configuration and internal instruction information;

[0134] 4. The computation acceleration management module issues subtask configurations, the routing management module configures the data routing for subtask execution, and the computation acceleration management module initiates the computation.

[0135] 5. The computing acceleration core performs calculations according to instructions, while the data plane performs data interaction and transmission according to configuration;

[0136] 6. Subtask complete. If all subtasks are completed, proceed to the next step; otherwise, return to step 4.

[0137] 7. The computation acceleration management module packages and returns the token computation result;

[0138] 8. Manage server / CPU update task information and output it;

[0139] 9. Return to 2 unless the model is cleaned up and exited.

[0140] In some embodiments, the large model system based on the computing accelerator card can also dynamically switch between over-limit working modes under the management of upper-layer software and the management CPU, adapting to a wider range of computing tasks. Specifically, when the large model system runs a continuous dialogue or a group of dialogues in the same domain for a preset time, the storage location of the weight parameters of all routing expert models is obtained, and the system working mode is adjusted according to the storage location;

[0141] First working mode: Minimal loss of precision, achieving stronger computational performance.

[0142] When weight parameters are stored in DDR / LPDDR video memory, high-frequency routing expert models that have been activated a set number of times are stored in HBM / GDDR video memory. During calculation, the routing algorithm is not calculated, and the model weights of the high-frequency routing expert models are used directly for calculation.

[0143] The second working mode sacrifices less precision but achieves stronger computing performance.

[0144] When weight parameters are stored in external SSD storage, high-frequency routing expert models that have been activated a set number of times are stored in HBM / GDDR video memory. During calculation, the routing algorithm is not calculated, and the model weights of the high-frequency routing expert models are used directly for calculation.

[0145] Third operating mode: Sacrificing SSD lifespan for enhanced computing performance.

[0146] When the system is switched to the third working mode, key-value compression can be canceled during computation, and the original KV cache can be stored on SSD. This not only reduces the computational load at the model level but also eliminates the need to recalculate the key-value pairs of all previous tokens for each token calculation, significantly reducing the real-time computation load of MLA, especially in long-sequence token computation scenarios such as long conversations, document generation, document analysis, and RAG. Taking deepseek-v3 as an example, the single-token, single-layer KV cache size is 128*2*128*2 bytes = 65536 bytes, far exceeding 4KB, which can form an SSD block, conforming to SSD storage characteristics. This mode offers a significant performance improvement but involves relatively frequent SSD access.

[0147] The third operating mode described above offers higher computational accuracy and performance than the first and second operating modes. The second operating mode uses external SSD storage less frequently than the third, thus extending its lifespan. These over-limit adjustment strategies can be configured and implemented by the management server according to actual needs.

[0148] In summary, by constructing a collaborative architecture of heterogeneous storage media and intelligent routing networks, achieving dynamic data scheduling of multi-level storage systems through normalized on-chip networks, utilizing hybrid memory combinations to differentiate and match the bandwidth-capacity requirements of model parameters, and leveraging scalable interconnect protocols to overcome physical limitations of hardware resources, this approach reduces storage costs while ensuring computational performance. It is compatible with traditional architectures and can construct flexible computing topologies, breaking through single-machine limitations and systematically resolving the technical contradictions of insufficient memory capacity, hardware resource mismatch, and high expansion costs of multiple servers in large-scale model deployments.

[0149] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A large-scale model system based on a computing acceleration chip, characterized in that, The system includes P computing acceleration units with direct access to memory modules and external SSD storage, and Q management servers. The memory modules are hybrid memory (HBM, GDDR, DDR, and LPDDR), used with external SSD storage to access model parameters. The P computing acceleration units are connected to an inter-chip interconnect (ISI) system for inter-chip data exchange. The P computing acceleration units and Q management servers are each connected to an Ethernet switching system. The target management server sends instructions and tasks to the target computing acceleration unit via the Ethernet switching system and performs external data exchange. Each computing acceleration unit incorporates a computing acceleration chip, connects to external SSD storage via an SSD interface, and connects to the ISI system via a network port and an ISI port. Each computing chip includes a normalized on-chip network (SoC), an interconnect transmission system connected to the SoC, a storage control system, and several computing acceleration cores. The SoC reads model parameters from the target location based on the storage control system and sends them to the computing acceleration cores for computation and data storage. The normalized on-chip network interacts with the management server and other computing acceleration units based on the interconnection transmission system, reading and storing external model parameters and inter-chip model parameters. The normalized on-chip network includes a storage route, a computing route, M intra-chip normalization modules connected to the storage control system, N-2 inter-chip normalization modules connected to the interconnection transmission system, and at least 2 off-chip normalization modules. The storage route is connected to the M intra-chip normalization modules, at least 2 off-chip normalization modules, and the computing route, respectively, and is used to store and retrieve intra-chip model data, off-chip model data, and send and receive various model data to and from the computing route. The computing route is connected to the N-2 inter-chip normalization modules and the computing acceleration core, respectively, and is used to send and receive inter-chip model data, send various received model data to the computing acceleration core, and obtain output data. The storage route includes at least M+2 levels of vertically distributed node pipelines, and each level of the node pipeline contains M+2 routing nodes cascaded end-to-end to form a ring structure. Each routing node at each level is connected to an uplink node and a downlink node. It receives model data through the downlink node and sends model data through the uplink node. The top-level node pipeline connects to N-2 inter-chip normalization modules and at least 2 off-chip normalization modules through uplink and downlink nodes to send and receive inter-chip model data and off-chip model data. The bottom-level node pipeline connects to the computation route through uplink and downlink nodes. In the M+2 level node pipeline, the downlink nodes of the same level routing nodes are cascaded vertically according to depth to send inter-chip model data and off-chip model data into the computation route. The upstream nodes of the same level routing nodes are cascaded in reverse vertical depth to send the model data output by the calculated routing to the inter-chip normalization module and the external-chip normalization module.

2. The large model system based on a computing acceleration chip according to claim 1, characterized in that, The storage control system includes several NVMe / Memory Extension CXL DDR controllers and corresponding PCIe RC subsystems, several DDR / LPDDR / GDDR controllers, and several HBM controllers. When the chip enters RC master mode, the NVMe controller reads or writes back model data in the external SSD storage based on the PCIe RC subsystem. The DDR / LPDDR / GDDR controllers read or write back model data in DDR / LPDDR / GDDR according to instructions. The HBM controllers read or write back model data in HBM according to instructions.

3. The large model system based on a computing acceleration chip according to claim 1, characterized in that, The interconnection transmission system includes a PCIe EP subsystem, a high-speed Ethernet subsystem, an inter-chip interconnection transmission controller, and several high-speed interconnection channels. When the chip enters EP slave mode, the PCIe EP subsystem communicates with the target number of management servers; or it communicates with the target number of management servers based on the high-speed Ethernet subsystem. The inter-chip interconnection transmission controller divides and reassembles tasks and inter-chip model parameters according to the data volume, and performs inter-chip transmission through the several high-speed interconnection channels.

4. The large model system based on a computing acceleration chip according to claim 1, characterized in that, The computational routing is configured with a computational routing receiving network, including at least M+2 unidirectional node pipelines, S+1 data stream merge nodes, and a distribution routing node. The M+2 unidirectional node pipelines respectively receive model data output from the storage route and transmit the model data in a pipelined manner through the S+1 cascaded receiving nodes included therein. Among the first S-level receiving nodes of the M+2 unidirectional node pipelines, the receiving nodes at the same level are connected to the first S-level data stream merge node, and the model data is sent to the computational acceleration core through the first S-level data stream merge node. The S+1-level receiving nodes of the M+2 unidirectional node pipelines are jointly connected to the S+1-level data stream merge node, and the S+1-level data stream merge node distributes the model data to N-2 inter-chip normalization modules through the distribution routing node.

5. The large model system based on a computing acceleration chip according to claim 4, characterized in that, The computational routing is configured with a computational routing transmission network, including a vertically distributed node pipeline ring of up to S+N-2 levels, a collection routing node and a distribution routing node. Each level of the node pipeline ring contains S+N-2 routing nodes that are cascaded end to end to form a ring structure. Each level of routing node is connected to an uplink node and a downlink node. Model data is received through the downlink node and model data is sent through the uplink node. The top-level node pipelined ring connects to N-2 inter-chip normalization modules and S ports connected to the computing acceleration core via uplink and downlink nodes to send and receive various model data. The bottom-level node pipelined ring connects to the computing route via uplink nodes and to the aggregation route node via downlink nodes. In the S+N-2 level node pipelined rings, the downlink nodes of the same level routing nodes are cascaded vertically in depth to send various model data to the aggregation route node. The uplink nodes of the same level routing nodes are cascaded vertically in reverse in depth to send the model data received by the computing route from the network output to the inter-chip normalization module and the computing acceleration core.

6. The large model system based on a computing acceleration chip according to claim 1, characterized in that, The computing acceleration chip also includes a data scoring module, a routing management module, and a model computing management module. The model computing management module allocates computing resources, storage resources, and bandwidth resources based on model characteristics, system hardware resources, and working mode settings. The overall strategy is set by an external management server based on the system situation, and the model computing management module within the chip translates the strategy into hardware configuration for execution. The routing management module configures the transmission of internal nodes of the normalized on-chip network according to the configuration information to obtain the transmission network that needs to be configured. These configurations can be dynamically modified to adapt to different tasks continuously. The data scoring module scores the data types processed by the system and the types and times of activated expert models to determine the model optimization decision for the chip response. Different optimization decisions activate different expert models.

7. The large model system based on a computing acceleration chip according to claim 6, characterized in that, The data scoring module includes an information reading probe unit, an information transmission pipeline, several scoring condition statistical threads, and a statistical analysis unit. The information reading probe unit detects the reading of the weight matrix by the computing acceleration core in real time. Each set of information is transmitted to each sub-conditional statistics thread through the information transmission pipeline. Different sub-conditional statistics threads collect different information according to the configuration conditions, and count the number of times the computing acceleration core reads the target route expert weights within a preset time. The statistical analysis unit converts the statistical information into a report based on computing resources. The report contains the topk of frequently read route expert weights for a single task, aggregates the topk of frequently read route expert weights for several tasks, and counts the percentage of topk reads in the total. The converted report is used for decision optimization calculations to activate or replace expert models and expert weights.

8. The large model system based on a computing acceleration chip according to claim 7, characterized in that, When the large model system runs a continuous dialogue or a group of dialogues within the same domain for a preset time, it obtains the storage location of the weight parameters of all routing expert models and adjusts the system's working mode according to the storage location. When weight parameters are stored in DDR / LPDDR video memory, the high-frequency routing expert models whose activation frequency reaches a set number are stored in HBM / GDDR video memory, and the system is adjusted to the first working mode. During calculation, the routing algorithm is not calculated, and the model weights of the high-frequency routing expert models are used for calculation. When weight parameters are stored in external SSD storage, the high-frequency routing expert models whose activation frequency reaches a set number are stored in HBM / GDDR video memory, and the system is adjusted to the second working mode. During calculation, the routing algorithm is not calculated, and the model weights of the high-frequency routing expert models are used for calculation. When the system is adjusted to the third working mode, all model parameters in the system are not compressed and are stored in external SSD storage. The third working mode has higher computational accuracy and performance than the first and second working modes, while the second working mode uses external SSD storage less frequently than the third working mode.

Images