Large model system based on calculation acceleration chip

Abstract

The invention discloses a large model system based on a calculation acceleration chip, and relates to the field of large models. Comprising a plurality of calculation acceleration units and a management server, and an inter-chip and off-chip data interaction network is formed and realized; a mixed video memory of the calculation acceleration unit is matched with an SSD to form a multi-source storage mode; a normalized network-on-chip, an interconnection transmission system, a storage control system and a plurality of calculation acceleration cores are arranged in the calculation chip; the normalized network-on-chip can read model parameters of a target position based on the storage control system and send the model parameters to the calculation acceleration core for calculation and recovery; and interacting with the management server and other computing acceleration units based on the interconnection transmission system, and reading and storing external model parameters and inter-chip model parameters. The technical problems of insufficient video memory capacity, too high hardware cost and limited transmission bandwidth in a traditional scheme are effectively solved through collaborative design of a mixed video memory architecture and a normalized network-on-chip in combination with a multi-stage routing control and dynamic configuration mechanism.

Description

Technical Field

[0001] The present application relates to the field of large models, and in particular to a large model system based on a computing acceleration chip. Background Art

[0002] Large models that meet commercial requirements generally require models with hundreds of billions of parameters. Considering the memory bottleneck of large models, the parameters are stored in full video memory (early GPUs were used as graphics cards, so all memory directly accessible to the acceleration chip was called video memory), and the mainstream method is to use full HBM storage. Due to the limited video memory of a single acceleration device (single card / single GPU), models with hundreds of billions of parameters often require multiple multi-card servers for inference. Figure 1 The typical 8-card server architecture shown shows that multi-server architectures incur high server interconnection construction costs. Furthermore, multi-server inference systems are challenging to develop in software, making them prohibitively expensive for small and medium-sized enterprises. The introduction of the H200 / H20 141GB graphics memory versions raises the upper limit of graphics memory capacity for a single server, enabling full graphics memory solutions for models like the llama3-405B and deepseek-v3-671B to be implemented on a single server, reducing system costs. However, the integration of dense HBMs places higher demands on the processing complexity of computing devices and chips.

[0003] The use of MOE architecture in large models, such as the Deepseek-v3 model, can greatly save operating costs. Its open source nature not only improves performance and reduces operating costs, but also makes low-cost all-in-one machines possible. Figure 2 The following figure shows a typical all-in-one architecture for the MOE architecture. The parameters for the MLA in this all-in-one are still stored in the GPU's GDDR (consumer-grade graphics cards use GDDR, while data center-grade graphics cards use HBM, which has higher bandwidth but also higher cost). The parameter design for the MLP (Multi-Layer Perceptron) takes advantage of the reduced bandwidth-to-capacity ratio required by the MOE model, replacing expensive HBM / GDDR with DDR. For example, using the 256 expert models in the deepseek-v3 model, a routing algorithm selects eight for each computation, reducing the bandwidth-to-capacity ratio by 1 / 16. However, this all-in-one design is a temporary solution. DDR is mounted on the CPU, limited by the 16-lane PCIe communication bandwidth between the CPU and GPU. If the MLP weights in DDR are transferred to the GPU for computation, the bandwidth-to-capacity ratio will be further significantly reduced. If the bandwidth-to-capacity ratio is maintained, the MLP computation must be performed on the CPU, significantly reducing computing power. Therefore, while the all-in-one architecture is cost-effective, it faces scalability challenges and a certain degree of mismatch in hardware resources.

[0004] Neither of the above solutions addresses the issue of KV cache expansion for exascale models. Taking the DeepSeek-v3 model as an example, the base model has 61 layers (excluding the MTP layer), with 128 heads per layer, each with a key and value dimension of 192 and a value dimension of 128. The data format is 2 bytes, BF16. Considering the default maximum sequence length of approximately 160KB, the KVcache space required for a single task is (192 + 128) * 2 bytes * 128 heads * 61 layers * 160KB, which is approximately 800GB. Compared to the 80GB memory of a mainstream data center-grade GPU and the 24-48GB memory of a consumer-grade GPU, this represents a significant amount of memory for a single task, and the cost soars with multiple tasks. Failure to use KVcache can lead to a surge in computational complexity. On the algorithmic side, the DeepSeek-v3 model introduces the Lora algorithm to compress the storage capacity required for the KV cache, balancing storage requirements with computing performance. Specifically, key and value storage is replaced by an intermediate compressed KV shared by 128 heads. During calculations, the key and value are multiplied by the corresponding weights in real time. The required storage after compression is 576 * 2 bytes * 61 layers * 160KB, approximately 11GB, an order of magnitude reduction in storage requirements. However, as the length of the sequence increases, this algorithm requires recalculating the key and value of the previous token in real time for each token. This increases the computational complexity of MLA and significantly degrades performance. Summary of the Invention

[0005] The embodiment of the present application provides a large model system based on a computing acceleration chip, which has the advantages of reducing hardware costs, improving storage bandwidth utilization, supporting flexible expansion architecture, and optimizing model parameter transmission efficiency compared to traditional model systems.

[0006] The large model system based on the computing acceleration chip includes P computing acceleration units with direct access to memory banks and external SSD storage, and Q management servers; the memory banks are a hybrid of HBM, GDDR, DDR, and LPDDR, and are used with external SSD storage to access model parameters; The P computing acceleration units are connected to the inter-chip interconnection switching system to perform inter-chip data exchange; the P computing acceleration units and the Q management servers are respectively connected to the Ethernet switching system, and the target management server sends instructions and tasks to the target computing acceleration unit through the Ethernet switching system and performs external data exchange; the computing acceleration chip is designed in the computing acceleration unit, connected to the external SSD storage through the SSD interface, connected to the inter-chip interconnection switching system through the network port, and connected to the inter-chip interconnection switching system through the inter-chip interconnection port; The computing chip is equipped with a normalized on-chip network, an interconnection transmission system connected to the normalized on-chip network, a storage control system, and several computing acceleration cores; the normalized on-chip network reads the model parameters of the target location based on the storage control system and sends them to the computing acceleration core for calculation and restoring model data; the normalized on-chip network interacts with the management server and other computing acceleration units based on the interconnection transmission system to read and restore external model parameters and inter-chip model parameters.

[0007] The beneficial effects of the technical solutions provided by the embodiments of the present application include at least the following: a system structure consisting of a management server and multiple computing acceleration units can distribute and accelerate large-scale model inference tasks in parallel, and cooperate with the inter-chip interconnection exchange system to realize the completion of subtasks and inter-chip data exchange. The core part of the computing unit, the computing acceleration chip, has a built-in storage control system, interconnection transmission system and normalized on-chip network. Through the coordinated design of the hybrid video memory architecture and the normalized on-chip network, combined with multi-level routing control and dynamic configuration mechanism, it effectively solves the technical problems of insufficient video memory capacity, high hardware cost and limited transmission bandwidth in traditional solutions. It has the advantages of reducing hardware cost, improving storage bandwidth utilization, supporting flexible expansion architecture and optimizing model parameter transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Typical 8-card server architecture shown; Figure 2 Shows the typical all-in-one structure of the MOE architecture; Figure 3 This is an overall structural diagram of a large model system based on a computing acceleration chip provided in an embodiment of the present application; Figure 4 A schematic diagram of another large-model computing acceleration chip architecture is shown; Figure 5 This is a schematic diagram of the configuration of the computing acceleration chip as a whole communicating with the outside world; Figure 6 A schematic diagram of the structure of a storage control system in a possible form is shown; Figure 7 The schematic diagram of the structure of the interconnected transmission system in one possible form is shown; Figure 8 shows a schematic diagram of a normalized on-chip network and its connection configuration; Figure 9 This is a schematic diagram of the network structure of the storage routing of this application; Figure 10 A schematic diagram of a network structure of an optimized storage routing is shown; Figure 11 Shown is a flow chart of the normalization and denormalization pipeline operations performed by various normalization modules; Figure 12 Shows a schematic diagram of the structure of the calculation routing receiving network; Figure 13 Schematic diagram of the structure of the calculation routing transmission network is shown in FIG; Figure 14 shows a schematic diagram of data processing of a normalized on-chip network; Figure 15 A schematic structural diagram of a routing management module in a possible form is shown. DETAILED DESCRIPTION

[0009] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0010] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0011] Figure 3 This is the overall structural diagram of the large model system based on the computing acceleration chip provided by the embodiment of the present application. The system includes P computing acceleration units, Q management servers, an inter-chip interconnection switching system, and an Ethernet switching system. The P computing acceleration units are connected to the inter-chip interconnection switching system to realize inter-chip data interaction between any two computing acceleration units, or one-to-many. The P computing acceleration units and Q management servers are respectively connected to the Ethernet switching system, and data communication between any management server and any other management server and computing acceleration unit is realized through the switching network. For the task of performing model inference calculation, the target management server sends instructions and tasks to the target computing acceleration unit through the Ethernet switching system and performs data exchange.

[0012] It should be noted that the management server in this application can be a management server or management CPU in a broad sense that is located on the upper layer of the accelerator card and has higher scheduling authority, which is used to assign tasks and transmit necessary model parameters. The management CPU relies on the management server and actually undertakes management tasks, but there are usually multiple management CPUs on a management server. For certain scenarios with low load requirements, a physical CPU can be virtualized into several or even more than a dozen CPUs as management terminals. In the above design, the model data can be stored in the SSD of each computing unit. In this way, when importing the model, each computing unit can import the weights from the local to the video memory, which not only shortens the path, but also can be executed in parallel, resulting in shorter latency.

[0013] The inter-chip interconnection switching system in the embodiment of the present application can be implemented using the NVLink switching chip + NV Switch network architecture, or other high-speed transmission protocols can be used as supplementary protocol designs. This technology NVLink has become a mainstream choice, especially in the training of ultra-large-scale models, so this application will not go into too much detail about it. The Ethernet switching system can use NICs network interface controllers to connect to single-layer or multi-layer Ethernet switches, and then the Ethernet switch communicates with the management server. For example, the NVIDIA Spectrum-X platform supports Scale-up (interconnection of accelerator cards within a single node) and Scale-out (interconnection between multiple nodes) through an optimized Ethernet architecture to meet the communication needs of large-scale models. This system structure is a commonly used network architecture, and this application will not go into too much detail about it.

[0014] The computing acceleration unit is based on the computing acceleration card and is connected to several external SSD storage devices through the SSD port, while the memory group is usually located inside the computing acceleration card. Figure 4 The figure shows the architecture diagram of the computing acceleration card. Each computing acceleration chip has several memory groups configured according to needs. The computing acceleration chip is designed in the computing acceleration unit, connected to the external SSD storage through the SSD interface, and connected to the inter-chip interconnection switching system through the high-speed network port. The high-speed inter-chip interconnection port connects to the inter-chip interconnection switching system to realize the communication between the acceleration card and the external data.

[0015] The memory group in this application is composed of a mixture of multiple storage devices, including but not limited to a hybrid memory composed of HBM, GDDR, DDR, and LPDDR, which is used together with external SSD storage to access model parameters (including KV Csche, token, various expert model weight parameters, etc.).

[0016] Figure 5 This is a schematic diagram of the overall architecture of a computing acceleration chip. The chip is divided into a compute plane and a data plane. The data plane includes a normalized on-chip network, an interconnect transmission system connected to the normalized on-chip network, and a storage control system. The compute plane integrates several compute acceleration engines / cores for model acceleration and high-concurrency processing of vector, scalar, and matrix data.

[0017] The storage control system is a controller cluster that supports the collaborative operation of multiple storage protocols. Based on various built-in storage controllers and their related protocols, it directly accesses the memory group and external SSD storage through the corresponding storage controller connection to access model parameters. For example, it is implemented using a combination of NVMe / memory expansion CXL controllers, PCIe systems, and HBM / GDDR / DDR independent controllers, allocating storage address space through an interleaved pattern. Through differentiated controller configuration, the system allocates parameters with the highest bandwidth-to-capacity ratio requirements to HBM memory, and parameters with lower requirements are allocated to DDR / LPDDR or even external SSDs. The compressed KV cache or original KV cache required for long texts is stored in external SSD storage to balance bandwidth requirements and storage costs.

[0018] The memory group in this application is a hybrid memory composed of heterogeneous storage media such as HBM, GDDR, DDR, and LPDDR. This is specifically implemented using a physical layer interface-compatible design and logical layer interleaved addressing technology. This combination reduces performance bottlenecks associated with single storage media by matching model parameter access patterns with media characteristics. HBM carries parameters for high-bandwidth-to-capacity demand models such as multi-head attention (MLA) calculations, while DDR / LPDDR storage routing expert models and compressed KV caches carry parameters for low-bandwidth-to-capacity demand models.

[0019] The normalized on-chip network is an intelligent routing architecture for unified management of intra-chip, inter-chip and off-chip data transmission paths. The interconnected transmission system is an important module for data communication with off-chip devices. Specifically, on-chip and off-chip data transmission channels that support multi-protocol communication can be designed, such as PCIe endpoint devices, high-speed Ethernet controllers and programmable inter-chip interconnect modules, and compatible with traditional acceleration devices through the protocol conversion layer. The system breaks through the physical limitations of a single machine with eight cards by dynamically splitting and reorganizing data streams, realizes flexible configuration of computing topology, and supports distributed access of model parameters in acceleration chip clusters. This application regards the computing accelerator card as a whole, and its data interaction and acceleration operations can be considered at the following three levels: 1. On-chip acceleration; When the computing acceleration card of the present application performs an acceleration task as a whole, the normalized network reads the on-chip model parameters of the target location based on the storage control system and sends them to a specific computing acceleration core for calculation, and stores the on-chip model data back.

[0020] 2. Off-chip acceleration: When the computing accelerator card needs to collaborate with an external management server / CPU, the normalized on-chip network interacts with the external management server off-chip based on the interconnection transmission system, receives instruction tasks, and reads and stores external model parameters.

[0021] 3. Inter-chip acceleration: When a computing accelerator card needs to exchange data with other computing accelerator cards in the system, the normalized on-chip network also interacts with other computing accelerator chips based on the interconnection transmission system to read and restore inter-chip model parameters.

[0022] In this embodiment, on-chip, inter-chip, and off-chip model data can be original tokens, KV Cache generated during the calculation process, various model weight parameters, intermediate data generated by multi-task parallel calculations, and calculation results, etc. The specific type is determined according to the task and different stages.

[0023] like Figure 5 As shown, in some embodiments, the computing acceleration chip may also have a built-in management plane, specifically including a routing management module, which can dynamically optimize the routing data transmission path of the normalized on-chip network and schedule system bandwidth resources based on the received instruction tasks. The accelerator card first allocates parameters to different storage media based on model characteristics and computing tasks. The MLA parameters that are accessed frequently are stored in HBM, the expert model parameters that only access part of the data each time are stored in DDR / LPDDR, and large-capacity data are stored in external SSD storage. During calculation, the normalized on-chip network reads the corresponding parameters from the hybrid video memory according to task requirements, and sends them to the computing acceleration core after preprocessing. For parameters that need to be accessed across chips, they are efficiently transmitted through the interconnected transmission system. The calculation results are written back to the corresponding storage location through the storage route. The entire process is dynamically optimized by the intelligent routing management module to ensure maximum data transmission efficiency.

[0024] The above technical solution significantly increases the accelerator card's memory limit through the use of hybrid memory, enabling the computation of exascale models with less hardware and fully leveraging the cost-effectiveness improvements brought by MOE computing. The use of multiple parallel SSD direct access expands the available KV cache storage space in multi-computing chip scenarios, improving performance while reducing the number of computing chips required and lowering system TCO. Furthermore, while maintaining low cost in minimal systems (single accelerator chip), parallel SSD direct access expands DDR / LPDDR memory space, accelerating fully cached model weights and KV cache on the card, rather than using management CPU memory. This more rationally utilizes hardware resources and provides higher computing performance.

[0025] In the above structure, the collaborative management of hybrid graphics memory and external SSD storage has problems such as interface protocol differences, access mode conflicts, and insufficient bandwidth utilization, resulting in low efficiency and limited flexibility in model parameter access. Figure 6 In the possible implementation shown, the storage control system may be configured in detail as follows: The storage control system is configured to include several NVMe / memory expansion CXL DDR controllers and corresponding PCIe RC subsystems, several DDR / LPDDR / GDDR controllers, and several HBM controllers. The NVMe controller is specifically used to control SSD storage. Specifically, when the PCIe RC subsystem enters RC master mode, the NVMe controller reads or restores model data from external SSD storage. The DDR / LPDDR / GDDR controller reads or restores model data from the DDR / LPDDR / GDDR storage according to instructions; the HBM controller reads or restores model data from the HBM storage according to instructions.

[0026] Specifically, in the above configuration, the combination of the NVMe / memory expansion CXL DDR controller and the PCIe RC subsystem enables the chip to actively initiate SSD storage access in RC master mode. Using the NVMe controller's acceleration function, the improvement over traditional SSD reads is reflected in the following aspects: 1. The task queue is inside the controller instead of in memory, which shortens the read and write paths and reduces latency. 2. The data cache area is also located inside the controller, which also has a shorter access path and lower latency; 3. Since the real-time processing capability of normalized data requires less data buffer space, it is possible to integrate the data buffer into the controller; 4. SSD addresses are mapped to the unified addressing of the storage network through logical addresses. Compared with traditional logical addresses that are stored as a table in the file system, accessing SSD data first reads and writes the file system, and then reads and writes the actual data, which reduces the number of processes and makes access faster.

[0027] Access to DDR / LPDDR / GDDR video memory does not require reliance on the PCIe system. It is an on-chip data transmission and can be directly scheduled according to demand. This application does not limit or elaborate on the specifics.

[0028] Because traditional PCIe-based architectures are limited by the minimum system requirement of eight cards per machine, hardware resources cannot be dynamically allocated based on task requirements. Traditional protocols also struggle to balance compatibility, flexibility, and transmission efficiency, resulting in poor system scalability, high deployment costs, and inefficient cross-device data transmission. To address this, this application designs an interconnected transmission system to achieve flexible expansion and efficient transmission. Figure 7 The diagram shows the structure of the interconnection transmission system, which includes a PCIe EP subsystem, a high-speed Ethernet subsystem, an inter-chip interconnection transmission controller, and several multi-access high-speed interconnection channel subsystems, such as N channels or other numbers, which are set according to actual requirements.

[0029] The PCIe EP subsystem is designed to be compatible with traditional GPU / accelerator card designs, enabling communication between the accelerator chip and the management CPU via PCIe. The high-speed Ethernet subsystem is designed to overcome the eight-card minimum requirement for a single system when building computing systems. This allows for a more flexible CPU + computing chip topology to be constructed over the network. This allows for the optimal CPU to computing chip ratio to be configured based on the computing scenario, and this ratio can be dynamically allocated as task requirements change, maximizing hardware utilization and reducing deployment costs.

[0030] The PCIe EP subsystem corresponds to the PCIe RC subsystem. When the chip enters EP slave mode, it communicates data with the target number of management servers through the high-speed Ethernet subsystem. For example, it uses the PCIe 5.0 protocol to achieve a 64GT / s transmission rate. This subsystem retains the physical interface and protocol stack of traditional GPU / accelerator cards, allowing the accelerator card to be seamlessly connected to traditional computing servers.

[0031] The high-speed interconnect channel and the corresponding inter-chip interconnect transmission controller support high-speed inter-chip interconnection. This can be implemented using mature protocols such as NVLink, InfiniBand, and RDMA, or customized and optimized transmission protocols. The high-speed interconnect channel subsystem implements communication up to the transport layer. The inter-chip interconnect transmission controller is responsible for both application-layer transmission protocols and the segmentation and reassembly of data for multi-channel transmission. For example, the inter-chip interconnect transmission controller has a built-in hardware-level data sharding engine that divides a single model parameter stream into multiple data blocks at a 512KB granularity and distributes them to different high-speed interconnect channels using a round-robin algorithm.

[0032] In traditional architectures, there's a lack of a unified path management mechanism between different types of storage controllers and external transmission subsystems. This results in low storage bandwidth utilization, redundant and inefficient on-chip and off-chip data transmission paths, and limited resource allocation flexibility when multiple controllers work together, making it difficult to achieve efficient access and dynamic scheduling of model parameters. To address this, this application introduces a normalized on-chip network and provides targeted improvements.

[0033] Figure 8 A schematic diagram of a normalized on-chip network and its connection configuration is shown. The normalized on-chip network includes storage routing, computation routing, M on-chip normalization modules connected to the storage control system, N-2 inter-chip normalization modules connected to the interconnection transmission system, and at least 2 off-chip normalization modules.

[0034] For convenience of description, the internal and external parts are divided into two groups A and B. Group A has M in-die normalization modules, corresponding to at most M storage controllers. Group B has N in total, of which at least one corresponds to a PCIe EP subsystem, at least one corresponds to a high-speed Ethernet subsystem, and the remaining at most N-2 correspond to inter-chip interconnection transmission controllers.

[0035] The number of in-die normalization modules corresponds to the maximum number of storage controllers. For the same type of storage controller, a single in-die normalization module can connect multiple controllers and optimize storage bandwidth through interleaving mode. For example, when two HBM controllers share the same module, the address space is divided into alternating access regions, and the controllers can access different data blocks in parallel, so that the total bandwidth reaches the theoretical peak of a single module. The off-die normalization module independently interfaces with the PCIe EP subsystem and the high-speed Ethernet subsystem. The former is compatible with the data packaging format of traditional PCIe devices, and the latter supports frame reassembly and verification of the Ethernet protocol stack. The inter-chip normalization module is directly connected to the interconnection transmission controller, and the model parameters are divided into fixed-size transmission units through predefined data segmentation rules, such as dividing the weight matrix by row into 512-byte data packets.

[0036] Since the traditional on-chip network cannot be compatible with the multi-source heterogeneous storage control strategy used in this application, the interaction between the multi-source heterogeneous storage controller and the inter-chip or off-chip module lacks an efficient pipelining scheduling mechanism, making it difficult to achieve efficient vertical depth transmission and dynamic reorganization of model parameters. Therefore, this application splits the on-chip network into two parts: storage routing and computing routing.

[0037] The storage routing is connected to the M in-die normalization modules, at least two off-die normalization modules, and the computing routing, respectively, for accessing in-die model data, off-die model data, and transmitting various model data to the computing route. When accessing in-die model data, the storage routing assigns priorities according to controller types, for example, HBM controller requests are prioritized over DDR controller processing, while the access requests of multiple controllers are executed in an interleaved manner through a pipeline mechanism. When accessing off-die data, the storage routing selects the corresponding off-die normalization module according to the protocol type, for example, Ethernet data passes through a dedicated CRC verification unit and enters the buffer queue. When sending data to the computing routing, the storage routing adds a header identifier of a unified format to data packets from different sources, for example, using 3 bits to identify the data source type and 2 bits to identify the target computing core number.

[0038] The compute router connects to N-2 inter-chip normalization modules and the compute acceleration core, respectively, for sending and receiving inter-chip model data, feeding the received model data into the compute acceleration core, and obtaining output data. Specifically, when processing on-chip model data access, the storage router receives access requests from the storage controller via the on-chip normalization module. For example, requests from four HBM controllers are merged into two on-chip normalization modules, each using a dual-channel interleaved mode, increasing the effective bandwidth to 1.8 times that of a single module.

[0039] In traditional on-chip routing architectures, the hierarchical design can lead to complex data transmission paths and insufficient bandwidth utilization. Interactions between multi-source heterogeneous storage controllers and inter-chip or off-chip modules lack an efficient pipeline scheduling mechanism, making it difficult to achieve efficient vertical and deep transmission and dynamic reorganization of model parameters. To address this issue, this application provides an improved storage routing architecture.

[0040] Figure 9 This is a schematic diagram of the network structure of the storage routing of this application. The storage routing includes at least M+2 levels of vertically distributed node pipeline rings. Each level of the node pipeline ring contains M+2 routing nodes cascaded end-to-end to form a ring structure. Each level of routing node is connected to an upstream node and a downstream node. The corresponding routing node receives model data transmitted by its respective downstream node and sends the model data through the upstream node.

[0041] For storage routing, considering the inter-data source exchange capacity matching of the storage routing normalization modules, a maximum of M+2 downstream pipeline rings are designed (more than M+2 is acceptable, but the system will not reach full bandwidth). However, in real-world scenarios, it is difficult to achieve the designed transmission limit, so considering limited area resources, this number can be reduced appropriately. The ring transmission method of the routing node pipeline supports data sharing bandwidth replication and distribution to any other normalization module, saving data exchange resources. Multiple parallel rings prevent congestion caused by large-volume transmission on other transmissions, which is particularly beneficial for transmissions with high latency performance.

[0042] The top-level node pipeline ring connects N-2 inter-chip normalization modules and at least 2 off-chip normalization modules through the upstream node and the downstream node, and sends and receives inter-chip model data and off-chip model data. The bottom-level node pipeline ring connects and calculates the routing through the upstream node and the downstream node, and sends or sends the model data.

[0043] In the M+2-level node pipeline ring, the downstream nodes of the routing nodes at the same level are cascaded according to the vertical depth pipeline, and the inter-chip model data and off-chip model data are sent to the calculation routing; the upstream nodes of the nodes at the same level are cascaded according to the vertical depth reverse pipeline, and the model data output by the calculation routing is sent to the inter-chip and off-chip normalization modules.

[0044] The top-level pipeline ring receives model data from the off-chip normalization module via downstream nodes and passes the data to the next-level pipeline ring. Routing nodes at the same level in each pipeline ring are connected vertically in order of depth, forming a unidirectional transmission link from top to bottom. When data arrives at the bottom-level pipeline ring, it is input into the computational routing via the downstream nodes for processing. The computation results are then transmitted back through the upstream nodes of the bottom-level pipeline ring in a reverse flow direction, ultimately distributed through the top-level pipeline ring to inter-chip or off-chip normalization modules. The vertical pipeline cascade structure allows multi-source data to be split into fixed-size data blocks during transmission. For example, each data block can be set to 128 bytes to match the burst transmission length of the HBM controller. The redundant design of the ring routing nodes ensures data transmission continuity through adjacent nodes even if a single node fails. The bandwidth differences between different memory controllers are accommodated by dynamically adjusting the pipeline cascade depth. For example, for a lower-bandwidth GDDR controller, the number of vertical pipeline stages can be increased to extend the data transmission window.

[0045] In some other embodiments, since the M+2 node pipeline ring fully supports data transmission modes under various conditions, it would be wasteful to configure an uplink node and a downlink node for each routing node. Figure 10 A schematic diagram of the network structure of an optimized storage routing is shown. The storage routing also includes at least M+2 levels of vertically distributed node pipeline rings. Each level of the node pipeline ring includes M+2 routing nodes that are cascaded end to end to form a ring structure, forming a node matrix. The top-level routing nodes in the node matrix connect to N-2 inter-chip normalization modules and at least two off-chip normalization modules, transmitting and receiving inter-chip and off-chip model data. The bottom-level routing nodes connect to the computational routing. Routing nodes in the node matrix are each configured with an upstream port and a downstream port. The downstream ports of routing nodes in the same column are pipelined vertically and depth-wise, feeding inter-chip and off-chip model data into the computational routing. The upstream ports of routing nodes in the same column are pipelined vertically and depth-wise inversely, feeding the model data output by the computational routing into the inter-chip and off-chip normalization modules.

[0046] In the above structure, the uplink and downlink nodes of each routing node are omitted, and all data transmission and forwarding tasks are transferred to the routing node for execution. Compared with the previous solution, each routing node originally designed 4 ports, while this solution adds 2 more, using 6 ports to implement a complete transmission solution, saving system scale and cost investment, while not affecting the overall performance with the support of sufficient pipeline quantity.

[0047] The above structure sets up a normalization module in the normalized on-chip network, but in fact, the model data from different sources have different formats, inconsistent compression states, and insufficient dynamic adaptation of storage and computing resources, resulting in low data transmission efficiency, increased processing delay of computing acceleration cores, and difficult to effectively alleviate the storage pressure of inter-chip and off-chip data transmission. Figure 11 It is shown in FIG that various normalization modules need to include normalization pipeline operations and inverse normalization pipeline operations.

[0048] When the chip performs normalized pipeline operations, it parses the received model data packets, determines the target computing acceleration core, synchronizes the parsed data based on the target computing acceleration core, decompresses the data according to the data status, converts the decompressed data to the target format, reorganizes the converted data according to the cutting dimensions, and then packages the reorganized data and sends it to the downstream node; When the chip performs the inverse normalization pipeline operation, it parses the received model data packet, determines the target computing acceleration chip or target management server, pipelines the data, converts the pipelined data into the target format, compresses the converted data according to the instructions, splits the compressed data according to storage tasks and types, and then packages the split data and sends it to the upstream node.

[0049] The six sequential execution steps of normalized pipeline operations can be summarized as follows: The first step is to determine the physical address of the target computing acceleration core by extracting the packet header identifier when parsing the model data packet; The second step, data synchronization, is to adapt to multiple storage controllers. If the storage controllers and normalization modules correspond one to one, this step is not required. In the third step, decompression selects the corresponding decoder according to the data status flag, for example, when the data packet is marked as LZ4 compression format, a dedicated hardware decoding unit is called; The fourth step is format conversion, which converts the data into the precision type supported by the target kernel. For example, INT8 quantized data is dequantized into FP16 format. In the fifth step, data reorganization dynamically adjusts the matrix block size based on the cutting dimension parameter. For example, the input tensor is split into 128×128 blocks to adapt to the parallel processing units of the computing core. Step 6: Repackage the data and send it to the downstream node.

[0050] The inverse normalization pipeline operation includes six reverse processing steps. The pipeline processing converts the calculation results into a continuous data stream, such as arranging the scattered matrix blocks in row-first order; the compression step selects the compression algorithm based on the characteristics of the storage medium, such as using a lossless compression algorithm for SSD storage; data segmentation dynamically allocates block size according to the storage task type, such as segmenting the returned data into 4KB aligned segments to adapt to the block storage characteristics of the SSD.

[0051] In short, whether it is normalization or denormalization, the purpose is to convert discrete storage addresses into unified access addresses; switch the data packet format, for example, block storage needs to form multiple packets of exactly 4KB; pack / reorganize multiple block data; data format conversion, command format conversion, such as converting the read and write commands of the storage address into the read and write commands of the NVMe logical block; adapt the computing core architecture, reorganize the data to adapt to the design of the internal systolic array and sparse matrix calculation of the computing core, so that data can be continuously input and sent out of the computing core.

[0052] If compression is used in the above scheme, the data must be compressed / decompressed during the pipeline. If the data is not compressed, the compression / decompression steps can be bypassed during the pipeline processing. For multidimensional data packet reassembly, in addition to the data structure, data padding may occur. Data padding is based on the characteristics of matrix calculations and generally uses zeros instead of directly filling all bits with zeros.

[0053] Regarding the structure of the computational routing, since it involves data transmission of multi-source heterogeneous storage, the design process of a single routing network is too complicated for complex situations with multiple concurrencies and high bandwidth. Therefore, this application chooses to further split the computational routing into a computational routing receiving network and a computational routing sending network.

[0054] Figure 12 The following diagram shows the structure of the compute routing receiving network. Factors affecting the compute routing receiving network include the highest traffic demand from graphics memory to the compute cores, which accounts for the largest proportion of transmissions; the relatively small amount of data returned from the compute memory or external management CPU; and the significantly smaller traffic from graphics memory to inter-chip communication than from storage to the compute cores. In terms of bandwidth, the theoretical bandwidth of graphics memory is much greater than that of inter-chip communication. Therefore, each port from the storage routing can directly reach each compute core / computing engine through a parallel pipeline. Data flows arriving at the normalization module are merged into one path and then implemented through one-to-many routing.

[0055] Specifically, the computational routing receiving network configuration includes at least M+2 receiving node pipelines, S+1 data stream merge nodes, and a distribution routing node. The receiving node pipeline is configured as a linear structure rather than a ring structure, and each receiving node pipeline contains S+1 receiving nodes cascaded end to end.

[0056] The M+2 receiving node pipeline receives various model data output by the storage router and transmits the model data in a pipeline through the S+1 end-to-end cascaded receiving nodes. Within the first S-level receiving nodes of this M+2 receiving node pipeline, the same-level receiving nodes are connected to the first S-level data flow merge nodes. Through the first S-level data flow merge nodes, the model data is sent to the computation routing transmission network, which ultimately routes and forwards it to the receive ports of all computation acceleration cores.

[0057] The S+1th level receiving nodes in this M+2 receiving node pipeline are connected to the S+1th level data flow merge node. The S+1th level data flow merge node distributes the model data to the computation routing and sending network through the distribution routing node. The computation routing and sending network ultimately routes and forwards the data to the receiving ports of the N-2 inter-chip normalization modules.

[0058] The number of receiving nodes depends on the number of receiving ports across all compute acceleration cores. For example, if each compute acceleration core has k input / output ports, for a total of S input / output ports, then S+1 receiving nodes are required, with the last receiving node dedicated to small amounts of inter-chip data transmission. Each merge node can aggregate model data from up to M+2 different sources, achieving high concurrency and redundancy.

[0059] During the first S stages of processing, each transmission unit simultaneously outputs processed data to the data flow merge node at the same level. For example, when S = 4, the outputs of the first four transmission units enter four independent merge nodes. The data flow merge node merges data from multiple receiving node pipelines at the same stage. For example, it combines the second-stage outputs of eight pipelines into a complete data block and sends it to the compute acceleration core. The remaining data after the first S stages of processing enters the S+1-level transmission unit. The final outputs of all receiving node pipelines are converged to the S+1-level data flow merge node. For example, when M+2 = 8, the final outputs of the eight pipelines are merged at this node. The distribution routing node dynamically distributes the integrated data to N-2 target modules based on the load status of the inter-chip normalization module, for example, selecting 14 of the 16 inter-chip modules that are currently idle for distribution. This structure reduces data latency through pipeline transmission, uses multi-level merge nodes to achieve data integration, and combines a dynamic distribution mechanism to improve task adaptability, enabling the computing acceleration core to simultaneously obtain multi-channel processed data while ensuring cross-chip data transmission efficiency.

[0060] The receiving node pipeline can be implemented using a FIFO structure, data flow merge nodes can be designed using multiplexers, and distribution routing nodes can distribute data based on lookup tables. This design enables the entire receiving network to efficiently process model data from different sources and dynamically adjust data flow based on demand.

[0061] Although the computational routing receiving network distributes model data to the computational acceleration core and the inter-chip normalization module, during the model data transmission process, how to efficiently integrate the data output by the computational acceleration core and the inter-chip interaction data and transmit them to the target module while ensuring the transmission bandwidth utilization and the flexibility of dynamic task allocation still has problems such as a single routing structure, limited transmission efficiency, and insufficient multi-path data integration capabilities.

[0062] This application further proposes Figure 13 The computational routing transmission network structure diagram shown in Figure 2 illustrates the computational routing transmission network. The external interconnect and computational acceleration cores experience significant traffic, while the computational acceleration cores return relatively little storage traffic. Therefore, high-speed interconnects are designed between the computational cores and between the computational cores and the external interconnect. Considering the ring network composed of multiple external computational acceleration chips (cards), efficient internal bypass forwarding and replication forwarding are implemented. A high-speed interconnect is also designed between the external interconnect ports, and data returning to the storage routing is sent serially.

[0063] Specifically, the computational routing sending network of the computational routing configuration includes at most S+N-2 levels of longitudinal distribution node pipeline rings, a collection routing node and a distribution routing node. Each level of the node pipeline ring contains (taking two off-chip normalization modules as an example) S+N-2 routing nodes that are cascaded end to end to form a ring structure. Each level of routing node is connected to an upstream node and a downstream node. The corresponding routing nodes receive the model data transmitted by their respective downstream nodes and send the model data through the upstream node.

[0064] The top-level node pipeline ring connects N-2 inter-chip normalization modules and S ports connected to the computing acceleration core through the upstream node and the downstream node, sending and receiving various model data. The bottom-level node pipeline ring connects the computing routing through the upstream node and the aggregation routing node through the downstream node.

[0065] In the S+N-2 level node pipeline ring, the downstream nodes of the routing nodes at the same level are cascaded in the vertical depth pipeline to send various model data to the aggregation routing nodes; the upstream nodes of the nodes at the same level are cascaded in the vertical depth reverse pipeline to send the model data output by the computing routing receiving network to the inter-chip normalization module and computing acceleration core.

[0066] The above-mentioned routing nodes are all designed for one-to-many or many-to-one transmission. For one-to-many transmission, each output channel uses a bit as a flag, and outputs when valid, which can support duplicate output; for many-to-one mode, polling is used for processing, which requires each input port to have a certain data buffering capacity, and at the same time set the polling priority and ratio, so as to have QoS management capabilities. For example, the downlink of each routing node can be configured as 8 parallel channels, supporting the transmission of 128 bits of data per cycle. The uplink realizes data distribution through reverse flow cascading, for example, using a priority polling mechanism to dynamically allocate transmission paths. The aggregation routing node can integrate multiple input buffer queues, each queue depth can be set to 64 data packets, and data integration is scheduled through a weighted polling algorithm. The distribution routing node can be configured to support dynamic port mapping tables, for example, adjusting the correspondence between ports and target modules in real time according to task requirements.

[0067] The computing routing sending and receiving network designed in this application realizes the efficient distribution of multi-source heterogeneous model data in computing routing. For the receiving network, firstly, the design of the multi-level receiving node pipeline improves the parallel reception capability of the model data and reduces the transmission delay. Secondly, the introduction of the multi-level data flow merge node ensures that the computing acceleration core can obtain multiple input data at the same time, avoiding idle resources. Finally, through the cooperation of the last-level merge node and the distribution routing node, the dynamic allocation of inter-chip data is realized, and the adaptability of the system to different task requirements is enhanced. This hierarchical and cascaded receiving network structure combined with the dynamic data flow integration mechanism effectively improves the data processing efficiency and flexibility of computing routing.

[0068] For the sending network, a multi-stage circular pipeline structure expands parallel transmission paths, avoiding the bandwidth limitations of a single channel. A vertical cascade mechanism optimizes timing synchronization and path selection flexibility, improving transmission efficiency. Dynamic allocation of aggregation and distribution nodes adapts to changing task requirements and enhances the ability to reconstruct multi-source data. This significantly improves the efficiency of integrated transmission of model data while ensuring the scalability and real-time performance of dynamic task allocation.

[0069] In some embodiments, Figure 10 Similarly, the computation routing and sending network can be further optimized by not using downstream and upstream nodes, that is, including at most S+N-2 levels of vertical distribution node pipeline rings, one aggregation routing node and one distribution routing node. Each level of the node pipeline ring contains S+N-2 routing nodes that are cascaded end to end to form a ring structure, forming a node matrix. The top-level routing nodes in the node matrix connect N-2 inter-chip normalization modules and S ports connected to the computing acceleration core to send and receive various model data. The bottom-level routing nodes are respectively connected to the computing routing receiving network and the aggregation routing node; the routing node connections in the node matrix are respectively provided with uplink ports and downlink ports. The downlink ports of the routing nodes in the same column are cascaded in a vertical depth pipeline to send various model data to the aggregation routing node; the uplink ports of the routing nodes in the same column are cascaded in a vertical depth reverse pipeline to send the model data output by the computing routing receiving network to the inter-chip normalization module and the computing acceleration core.

[0070] In the above design, all data forwarding and transmission between pipeline rings are transferred to the routing nodes for execution. Although this sacrifices the routing function of some routing nodes, it can fully achieve high-redundancy parallel forwarding when there are enough pipeline rings.

[0071] Based on the design network of the above storage routing and calculation routing, this application can summarize the following scenarios based on data type and execution content, see Figure 14 Schematic diagram of data processing is shown.

[0072] Step 1: Read the on-chip model data in the Menory memory group and external SSD storage through the on-chip normalization module and send it to the storage router; Step 2: The storage router sends the model data obtained from various sources to the routing calculation receiving network; Step 3: The routing calculation sending network sends the received model data to the routing calculation sending network; Step 4: The routing calculation sending network sends the received model data to the corresponding calculation acceleration core to perform related accelerated calculations; Step 5: The routing calculation sends the network to receive the output of the calculation acceleration core and the relevant model data; Step 6: The routing calculation sending network sends the received relevant model data (including the generated temporary data and calculation results, etc.) back to the storage router; Step 7: The storage router receives the relevant model data output by the routing calculation and sends it to the target storage location (which can be an external SSD storage and Menory video memory group); Step ⑧, the storage router receives instructions and tasks from the management server or management CPU through the off-chip normalization module, as well as off-chip model data related to the task; Step 9: The storage router sends / feeds back the relevant model data output by the routing calculation network to the management server or management CPU through the off-chip normalization module; Step ⑩: The routing calculation sending network receives the inter-chip model data sent by other computing acceleration chips through the inter-chip normalization module; Step ⑪: The routing calculation sending network sends the received relevant model data to other computing acceleration chips through the inter-chip normalization module.

[0073] Among all the above steps, the execution is carried out in combination according to the specific functions and tasks. For example, the steps of reading the model parameters from the external SSD storage and returning them to the management CPU after calculation should be steps ①+②+③+④+⑤+⑥+⑨. When it is necessary to receive the management CPU instruction and transmit the model data in the DDR to other computing acceleration chips, it should be steps ①+②+③+⑪. When it is necessary to obtain other computing acceleration chips to perform accelerated operations and store them in the SSD, it should be steps ⑩+④+⑤+⑥+⑦. The steps of the chip performing other different tasks are not repeated in this embodiment.

[0074] In some embodiments, the computing acceleration chip architecture implements access and transmission of model parameters through a normalized on-chip network, interconnected transmission system, and storage control system. However, during execution, there are problems such as rigid hardware resource allocation, the inability of the transmission network to dynamically adapt to different tasks, and the lack of a model optimization decision-making mechanism based on data type and usage frequency. This leads to insufficient resource utilization, limited task processing efficiency, and the inability to flexibly adjust computing strategies according to actual needs. Figure 4 As shown, the present application can also add a model calculation management module and a data scoring module to the management plane part of the computing acceleration chip.

[0075] Its model computation management module is configured to receive overall policies set by an external management server, translate them into hardware configuration instructions, and dynamically allocate computing, storage, and bandwidth resources based on the model's characteristics and the system's hardware resource status. For example, in a hybrid memory architecture, this module can prioritize the weights of frequently accessed expert models to high-bandwidth storage areas based on the bandwidth differences between HBM and DDR.

[0076] The routing management module is configured to dynamically modify the transmission configuration of nodes within the normalized on-chip network. By adjusting the cascading of node pipelines or transmission protocol parameters, the transmission network can adapt to the data transmission mode of different tasks. For example, when executing MOE model inference, the number of transmission channels of the inter-chip normalization module can be temporarily increased to improve the efficiency of expert weight distribution.

[0077] The data scoring module is configured to calculate and accelerate the frequency of weight matrix reads. Using a conditional statistics thread within the chip, it captures frequently accessed expert model types in real time and generates a statistical report containing the top-k routing expert weights. For example, if the number of expert model calls for a certain type of sparse activation pattern exceeds a preset threshold, an optimization decision is triggered to migrate the model weights to a low-latency storage area.

[0078] In a possible implementation, the routing management module may be configured as follows: Figure 15 The structure shown in the figure specifically includes an information reading probe unit, an information transmission pipeline, several conditional statistics threads, and a statistical analysis unit. The information reading probe unit detects the calculation acceleration core's reading of the weight matrix in real time. Each set of information is transmitted to each conditional statistics thread through the information transmission pipeline. Different conditional statistics threads collect different information based on the configured conditions, counting the number of target routing expert weights read by the calculation acceleration core within a preset time.

[0079] The statistical analysis unit converts statistical information into specific reports based on computing resources. The reports include the top k high-frequency readings of routing expert weights for a single task, the aggregation of the top k high-frequency readings of routing expert weights for several tasks, and the statistics of the percentage of top k readings in the total. The converted reports are used for decision-making optimization calculations, activating or replacing expert models and expert weights.

[0080] Specifically, the information reading probe unit can be configured to embed a hardware probe in the physical address access path of the weight matrix. When the computing acceleration core initiates a weight reading operation, the probe captures the target weight identifier, access timestamp, and associated task identifier. The information transmission pipeline can adopt a ring buffer structure. Each buffer unit stores triples or five-tuple information of a single weight reading event. The buffer depth is set to 1.5 times the maximum theoretical access rate of the computing acceleration core to ensure that no data loss occurs in high-load scenarios. The number of conditional statistical threads can be set to an integer multiple of the chip computing acceleration core. Each thread is bound to an independent memory workspace. The statistical conditions include but are not limited to task type, time window, and weight version number. The preset time window can be configured to range from 10 milliseconds to 1 second. The statistical analysis unit has a built-in sliding time window algorithm to aggregate the multi-threaded statistical results in the time dimension. The k value of topk is dynamically adjusted according to the currently available computing resources, and the value range is an integer between 5 and 20.

[0081] When the computing acceleration core executes the inference task, the read information probe unit records the weight access event in real time through a hardware-level signal capture mechanism to generate an original data stream containing a task ID, a weight ID, and a timestamp. The information transmission pipeline alternately receives the detected data using a double buffering mechanism and distributes the data to the memory work area of each statistical thread. The conditional statistical thread maps the weight ID to different statistical dimensions according to a preset hash rule, for example, when expert weights are classified and counted by task type, a Bloom filter is used to quickly screen the target data and count the access frequency in the statistical window. The statistical analysis unit performs global aggregation every 500 milliseconds to normalize the topk weights across tasks, calculate their proportion of the total access volume, and generate a visualization report containing a weight heat map and a time distribution curve. This report is input to the dynamic scheduling algorithm, which triggers the weight preloading mechanism when it detects that the access proportion of a certain expert weight exceeds 30% for three consecutive statistical periods, and migrates the high-frequency weight to the HBM video memory; when it finds that the call frequency of a certain expert model is below a threshold, it replaces it from the computing resource pool. Through a multi-stage pipeline and parallel statistical architecture, the extraction delay of weight access features is controlled within 5 milliseconds, which is more than 8 times more efficient than traditional polling statistical methods, and the response speed of dynamic resource allocation is improved by 3 times.

[0082] When the large model system of the present application is applied to an actual multi-storage hybrid video memory scene, the computing strategy first considers the storage of weight data. Taking the 671B model in deepseek-v3 as an example, only about 37B weights are activated for each token calculation, among which the MLA weights and the shared expert weights in the MLP are activated for each token calculation; and the routing expert weights are 600B+, but based on the MOE routing algorithm, only 1 / 32 are activated for each token calculation. Based on the above characteristics, in the design of the demonstration system, for a configured sub-computing system (acceleration card architecture), the dynamic weight and cache storage strategy can be designed as follows: 1. The number of computing cards is large, and the HBM / GDDR video memory capacity reaches TB level. Weight storage: all weights are stored in HBM / GDDR video memory; Kv cache storage: store compressed kvcache in DDR / LPDDR video memory.

[0083] 2. The number of computing cards is moderate, and the HBM / GDDR video memory capacity is insufficient, but the DDR / LPDDR video memory capacity is in the TB level. MLA weights are stored in HBM / GDDR memory; MLP weights are stored in DDR / LPDDR (if conditions permit, shared expert weights are preferentially stored in HBM / GDDR memory, and routing expert weights are stored as much as possible in HBM / GDDR memory); Kvcache is preferentially stored in DDR / LPDDR; if insufficient, it is stored in SSD.

[0084] 3. There are only a few or even only one computing card, and the HBM / GDDR memory and DDR / LPDDR memory capacity are insufficient; MLA weights are stored in HBM / GDDR memory; shared expert weights are stored in HBM / GDDR memory first, and in DDR / LPDDR memory second; routing expert weights are stored in DDR / LPDDR memory as much as possible, and in SSD second.

[0085] The subsystem calculation process corresponding to the accelerator card architecture is as follows: 1. The management server / CPU initializes the model according to resource configuration, manages the import of model weights into the corresponding storage space, and imports model information and calculation strategies into the model calculation acceleration management module inside the chip; 2. The management server / CPU determines the tokens to be calculated based on the task, converts them into embeddings, and organizes them into parallel input data (such as prefilling multiple consecutive tokens in a single conversation or generating a single token in parallel across multiple conversations). The input data is then downloaded to each computing acceleration chip. 3. The computing acceleration chip computing acceleration management module determines how to perform subtasks for token calculations in this batch based on the configuration (a single-layer decoder can be a subtask, each MLA / MLP can be a subtask, and there are multiple ways to divide it), and generates configuration and internal instruction information; 4. The computing acceleration management module issues subtask configurations, the routing management module configures data routing for subtask execution, and the computing acceleration management module initiates the computation. 5. The computing acceleration core performs calculations according to instructions, and the data plane performs data interaction and transmission according to configuration; 6. Subtasks are completed. If all subtasks are completed, proceed to the next step. Otherwise, return to step 4. 7. The calculation acceleration management module packages and returns the token calculation results; 8. Manage server / CPU update task information and output; 9. Return to 2, unless the model is cleaned and ejected.

[0086] In some embodiments, the large-scale model system based on the computing accelerator card can also implement dynamic over-limit working mode switching under the management of the upper-layer software and the management CPU to adapt to a wider range of computing tasks. Specifically, when the large-scale model system runs a continuous dialogue or a group of dialogues in the same field for a preset time, the weight parameter storage location of all routing expert models is obtained and the system working mode is adjusted according to the storage location; The first working mode: less loss of precision, better computing performance When there are weight parameters stored in the DDR / LPDDR memory, the high-frequency routing expert model whose activation frequency reaches the set number of times is stored in the HBM / GDDR memory. During the calculation, the routing algorithm is not calculated, and the model weight of the high-frequency routing expert model is directly used for calculation.

[0087] The second working mode has a small loss of precision and obtains stronger computing performance When weight parameters are stored in external SSD storage, the high-frequency routing expert model whose activation frequency reaches the set number of times is stored in HBM / GDDR memory. The routing algorithm is not calculated during calculation, and the model weight of the high-frequency routing expert model is directly used for calculation.

[0088] The third working mode: Lose the life of the SSD, but gain stronger computing performance When the system switches to the third operating mode, key and value compression is removed during calculations, and the original KV Cache is stored in the SSD. This not only reduces computational complexity at the model level but also eliminates the need to recalculate the keys and values ​​of all previous tokens for each token calculation. This significantly reduces the amount of real-time MLA computation required, especially for long-sequence token computation scenarios such as long conversations, document generation, document analysis, and RAG. For example, in deepseek-v3, the KV Cache size for a single token in a single layer is 128*2*128*2 bytes = 65536 bytes, significantly larger than 4KB and suitable for SSD blocks, which is consistent with SSD storage characteristics. This mode significantly improves performance, but requires relatively frequent SSD access.

[0089] The third operating mode offers higher computational accuracy and performance than the first and second operating modes. The second operating mode utilizes external SSD storage less frequently than the third operating mode, thus extending its lifespan. The above over-limit adjustment policies can be configured and implemented by the management server based on actual needs.

[0090] In summary, by constructing a cooperative architecture of heterogeneous storage media and intelligent routing network, dynamic data scheduling of multi-level storage system is realized through normalized network-on-chip, bandwidth-capacity requirement of model parameters is matched by using hybrid display memory, and physical limitation of hardware resources is broken through by using scalable interconnection protocol, so that the computing performance is guaranteed while the storage cost is reduced, the traditional architecture is compatible, the flexible computing topology is constructed, the single machine limitation is broken through, and the technical contradictions of insufficient display memory capacity, hardware resource mismatch and high expansion cost of multiple servers in large model deployment are systematically solved.

[0091] The specific embodiments are only an explanation of the present application, and are not a limitation of the present application. Those skilled in the art can make modifications to the embodiments without creative contribution after reading the specification, as long as the modifications are within the scope of the claims of the present application.

Claims

1. A large model system based on a computing acceleration chip, characterized in that: It includes P computing acceleration units with direct access memory groups and external SSD storage, and Q management servers; the memory groups are a mix of HBM, GDDR, DDR, and LPDDR, and are used with external SSD storage to access model parameters; The P computing acceleration units are connected to the inter-chip interconnection switching system to perform inter-chip data exchange; the P computing acceleration units and the Q management servers are respectively connected to the Ethernet switching system, and the target management server sends instructions and tasks to the target computing acceleration unit through the Ethernet switching system and performs external data exchange; the computing acceleration chip is designed in the computing acceleration unit, connected to the external SSD storage through the SSD interface, connected to the inter-chip interconnection switching system through the network port, and connected to the inter-chip interconnection switching system through the inter-chip interconnection port; The computing chip is equipped with a normalized on-chip network, an interconnection transmission system connected to the normalized on-chip network, a storage control system, and a plurality of computing acceleration cores; The normalized on-chip network reads the model parameters of the target location based on the storage control system and sends them to the computing acceleration core for calculation and restoring model data; The normalized on-chip network interacts with the management server and other computing acceleration units based on the interconnection transmission system to read and restore external model parameters and inter-chip model parameters.

2. The large model system based on a computing acceleration chip according to claim 1, characterized in that: The storage control system includes several NVMe / memory extension CXL DDR controllers and corresponding PCIe RC subsystems, several DDR / LPDDR / GDDR controllers, and several HBM controllers; When the PCIe RC subsystem enters the RC master mode, the NVMe controller reads or stores back the model data in the external SSD storage based on the PCIe RC subsystem; the DDR / LPDDR / GDDR controller reads or stores back the model data in the DDR / LPDDR / GDDR according to the instruction; and the HBM controller reads or stores back the model data in the HBM according to the instruction.

3. The large model system based on a computing acceleration chip according to claim 1, characterized in that: The interconnection transmission system includes a PCIe EP subsystem, a high-speed Ethernet subsystem, an inter-chip interconnection transmission controller, and several high-speed interconnection channel subsystems; The PCIe EP subsystem performs data communication with the target number of management servers when the chip enters the EP slave mode; or performs data communication with the target number of management servers based on the high-speed Ethernet subsystem; The inter-chip interconnection transmission controller divides and reorganizes tasks and inter-chip model parameters according to data volume, and performs inter-chip transmission through a plurality of the high-speed interconnection channels.

4. The large model system based on a computing acceleration chip according to claim 1, characterized in that: The normalized on-chip network includes a storage router, a computation router, M on-chip normalization modules connected to the storage control system, N-2 inter-chip normalization modules connected to the interconnection transmission system, and at least two off-chip normalization modules; The storage router is respectively connected to the M on-chip normalization modules, at least two off-chip normalization modules, and the computing router, and is respectively used to access on-chip model data, off-chip model data, and send and receive various model data to the computing router; The computing router is connected to the N-2 inter-chip normalization modules and the computing acceleration core, and is respectively used for sending and receiving inter-chip model data, sending the received various model data to the computing acceleration core and obtaining output data.

5. The large model system based on a computing acceleration chip according to claim 4, characterized in that: The storage routing includes at least M+2 levels of longitudinally distributed node pipeline rings, each level of the node pipeline ring includes M+2 routing nodes that are cascaded end to end to form a ring structure; Each level of routing node is connected to an upstream node and a downstream node, receiving model data through the downstream node and sending model data through the upstream node; The top-level node pipeline ring is connected to N-2 inter-chip normalization modules and at least 2 off-chip normalization modules through the upstream node and the downstream node, and transmits and receives inter-chip model data and off-chip model data. The bottom-level node pipeline ring is connected to the calculation route through the upstream node and the downstream node; In the M+2-level node pipeline ring, the downstream nodes of the routing nodes at the same level are cascaded in a vertical depth pipeline to send the inter-chip model data and the off-chip model data to the calculation routing; The upstream nodes of the routing nodes at the same level are cascaded in reverse flow according to the vertical depth, and the model data output by the calculation routing is sent to the inter-chip normalization module and the off-chip normalization module.

6. The large model system based on a computing acceleration chip according to claim 4, characterized in that: The computational routing is configured with a computational routing receiving network, comprising at least M+2 unidirectional node pipelines, S+1 data flow merge nodes, and a distribution routing node; the M+2 unidirectional node pipelines respectively receive the model data output by the storage routing, and transmit the model data in a pipeline manner through the S+1 head-to-tail cascaded receiving nodes contained therein; Among the first S-level receiving nodes of the M+2 unidirectional node pipelines, the same-level receiving nodes are respectively connected to the first S-level data flow merge nodes, and the model data is sent to the computing acceleration core through the first S-level data flow merge nodes; The S+1th level receiving nodes of the M+2 unidirectional node pipelines are commonly connected to the S+1th level data flow merge node, and the S+1th level data flow merge node distributes the model data to the N-2 inter-chip normalization modules through the distribution routing node.

7. The large model system based on a computing acceleration chip according to claim 6, characterized in that: The computational routing configuration includes a computational routing transmission network, including at most S+N-2 levels of vertically distributed node pipeline rings, a collection routing node, and a distribution routing node. Each level of the node pipeline ring includes S+N-2 routing nodes that are cascaded end to end to form a ring structure. Each level of routing node is connected to an upstream node and a downstream node, receives model data through the downstream node, and sends model data through the upstream node. The top-level node pipeline ring is connected to N-2 inter-chip normalization modules and S ports connected to the computing acceleration core through the upstream node and the downstream node to send and receive various model data. The bottom-level node pipeline ring is connected to the computing router through the upstream node and to the aggregation routing node through the downstream node. In the node pipeline ring of the S+N-2 level, the downstream nodes of the routing nodes at the same level are cascaded in a vertical depth pipeline to send various model data to the aggregation routing node; the upstream nodes of the routing nodes at the same level are cascaded in a vertical depth reverse pipeline to send the model data output by the computing routing receiving network to the inter-chip normalization module and the computing acceleration core.

8. The large model system based on a computing acceleration chip according to claim 1, characterized in that: The computing acceleration chip also includes a data scoring module, a routing management module, and a model computing management module; The model calculation management module allocates computing resources, storage resources, and bandwidth resources based on model characteristics, system hardware resources, and working mode settings. The overall strategy is set by an external management server based on system conditions, and the model calculation management module in the chip translates the strategy into hardware configuration for execution. The routing management module performs transmission configuration on the internal nodes of the normalized on-chip network according to the configuration information to obtain the transmission network that needs to be configured. These configurations can be dynamically modified to adapt to different tasks continuously; The data scoring module scores the data type according to the data type processed by the system and the type and number of activated expert models, and determines the model optimization decision of the chip response. Different optimization decisions activate different expert models.

9. The large model system based on a computing acceleration chip according to claim 8, characterized in that: The data scoring module includes an information reading probe unit, an information transmission pipeline, several conditional statistical threads, and a statistical analysis unit; The information reading probe unit detects the reading of the calculation acceleration check weight matrix in real time, and each set of information is transmitted to each conditional statistics thread through the information transmission pipeline; Different conditional statistics threads count different information according to configuration conditions, and count the number of target routing expert weights read by the calculation acceleration core within a preset time; The statistical analysis unit converts the statistical information into a report based on computing resources. The report includes the topk routing expert weights read frequently by a single task, aggregates the topk routing expert weights read frequently by several tasks, and counts the percentage of topk readings in the total. The converted report is used for decision optimization calculations, activation or replacement of expert models and expert weights.

10. The large model system based on a computing acceleration chip according to claim 9, characterized in that: When the large model system runs a continuous dialogue or a group of dialogues in the same field for a preset time, the weight parameter storage locations of all routing expert models are obtained, and the system working mode is adjusted according to the storage locations; When there are weight parameters stored in the DDR / LPDDR memory, the high-frequency routing expert model whose activation frequency reaches the set number of times is stored in the HBM / GDDR memory, and the system is adjusted to the first working mode. During calculation, the routing algorithm is not calculated, and the model weight of the high-frequency routing expert model is used for calculation; When there are weight parameters stored in the external SSD storage, the high-frequency routing expert model whose activation frequency reaches the set number of times is stored in the HBM / GDDR video memory, and the system is adjusted to the second working mode. The routing algorithm is not calculated during the calculation, and the model weight of the high-frequency routing expert model is used for calculation; When the system is adjusted to the third working mode, all model parameters in the system are not compressed and stored in the external SSD storage; The calculation accuracy and performance of the third working mode are higher than those of the first and second working modes, and the second working mode uses external SSD storage less frequently than the third working mode.

Images