A method and system for expert parallelism processing of a large language model based on MoE

Abstract

The application belongs to the field of machine learning, and discloses a MoE-based large language model expert parallel processing method and system. Through dynamically allocating expert quantization bit width and sparse mode, predicting and prefetching activated expert parameters, grouping token generation task queue and dynamically configuring hardware accelerator, efficient parallel processing of the MoE model is realized. First, based on the importance score calculated according to the expert historical activation frequency, weight distribution and model structure, the quantization precision and sparse proportion are adaptively allocated, so as to avoid resource waste and precision loss of the unified strategy. Second, the activated experts are predicted by using the current layer hidden state and historical activation sequence, the parameters are loaded into the dedicated cache in advance, the high-bandwidth memory access is reduced, and the bandwidth peak contention is relieved. Third, the tokens are grouped by using the token-expert mapping table, the task queue is created, the dynamic configuration of the systolic array is combined, the heterogeneity problem of the compressed experts is solved, and efficient parallel operation of the mixed precision matrix is ensured.

Description

Technical Field

[0001] This invention belongs to the field of machine learning technology, specifically relating to an expert parallel processing method and system for large-scale language models based on MoE. Background Technology

[0002] Large Language Models (LLMs) have profoundly changed the way many industries operate, playing a crucial role in natural language processing, content generation, and decision support. In natural language processing, they encompass scenarios such as text translation, sentiment analysis, and intelligent dialogue; for example, everyday translation software and customer service robots are supported by related technologies. Content generation includes article writing, creative copywriting, and code generation, with many creators using them to improve content production efficiency. In decision support, they help companies analyze market data, identify industry trends, and even assist doctors in organizing case information and providing preliminary diagnostic references in the medical field. Traditionally, these large language models are deployed in data centers equipped with high-end graphics processing units (GPUs) because model operation requires powerful computing capabilities. However, in recent years, more and more people want to run these models on consumer-grade platforms (such as personal computers, smartphones, tablets, and other devices commonly used by ordinary users). The main reason is that local operation better protects data privacy—data does not need to be uploaded to remote data centers, reducing the risk of leakage in transit—while also improving inference speed—avoiding latency during data transmission over the network, allowing the model to respond more promptly.

[0003] However, deploying large language models on consumer platforms faces a key challenge: their enormous memory requirements. Generally, large language models require hundreds of gigabytes of memory to run, while most consumer devices' graphics processing units (GPUs) typically only have tens of gigabytes of memory, far from meeting the model's needs. This memory mismatch directly leads to a significant drop in model performance, such as frequent stuttering, slow response times, and even failure to start due to insufficient memory. This not only impacts the user experience but also makes it difficult for personal devices to effectively utilize the capabilities of large language models, limiting the widespread adoption and application of this technology among ordinary users. Many users, even those with local usage needs, are unable to achieve this due to hardware memory limitations.

[0004] To alleviate the memory bottleneck of local deployment of large language models, various solutions have been proposed in the industry, among which the Mixture of Experts (MoE) architecture is a promising approach. The core of this architecture is to break down the originally large model into multiple small expert modules, each responsible for handling a portion of the model's tasks. During inference, it is not necessary to activate all expert modules; only a small subset relevant to the current task needs to be activated. This approach ensures that the overall accuracy of the model is not significantly affected while substantially reducing the computational resources and memory required at runtime, allowing the model to run on resource-constrained devices. With the application of the MoE architecture, further improving its efficiency and scalability has become a new focus, making hardware acceleration technology a key research direction. Solutions based on Field-Programmable Gate Arrays (FPGAs) have become a recent research focus due to their high flexibility, good energy efficiency, and suitability for customized sparse computation. Currently, there are many FPGA-based design solutions, such as the FLAME framework. FLAME is the first FPGA framework to fully utilize the MoE sparsity in Transformer (a commonly used deep learning architecture). It achieves this through parameter-level N:M structured pruning (pruning unimportant parameters in the model according to a fixed ratio, e.g., retaining M useful parameters out of every N parameters) and expert-level cyclic expert prediction (CEPR). Prediction (predicting upcoming expert modules to improve accuracy) optimizes model performance. A double-buffering mechanism is also introduced—allowing data loading and computation of expert modules to occur simultaneously, reducing waiting time and improving overall processing speed. There are also solutions optimized for multi-task scenarios, such as M3ViT, which uses zero-overhead task switching technology to dynamically activate the required sparse expert paths without wasting extra resources when switching between different processing tasks. Edge-MoE, on the other hand, maximizes the reuse of hardware resources and reduces waste through efficient Gelu approximation (Gelu is a commonly used activation function that reduces computation while maintaining performance) and a unified linear layer design.Furthermore, heterogeneous processing architectures are constantly innovating. For example, the Duplex architecture combines an eXtensible Processing Unit (XPU) with Processing-in-Memory (PIM, a technology that combines computing and storage to reduce the time it takes to move data between storage and computing units), allowing the expert processing phase and attention computation phase of the MoE model to run in parallel, further improving the utilization of computing resources and memory. The Space-Mate architecture, on the other hand, is designed for Simultaneous Localization and Mapping (SLAM) tasks on mobile devices, commonly used in drone navigation, augmented reality (AR) scenarios, etc. It achieves low latency and low power consumption inference processes by using a Sparse Mixture of Experts (SMoE) router and a dual-skip core architecture with a coarse-grained sparse pattern.

[0005] While existing technologies have made some progress in alleviating the memory and computational pressure on MoE models, several problems remain to be solved. Firstly, uniform pruning and quantization strategies are inefficient. Many current hardware acceleration solutions for MoE models apply a fixed pruning ratio and uniform quantization precision to all expert modules, ignoring the significant differences in activation frequencies among different expert modules—some are frequently used during inference, while others are rarely activated. This "one-size-fits-all" approach is prone to problems: for expert modules with high activation frequencies, excessive parameter pruning or quantization precision reduction may lead to decreased model processing accuracy, affecting the accuracy of the final result; conversely, for expert modules with low activation frequencies, retaining too many unnecessary parameters wastes valuable computational resources and memory. Secondly, there are issues caused by the delay in expert selection. During MoE model inference, the expert modules that need to be activated at each layer are dynamically changing, depending on the hidden state of the input token (the basic data unit processed by the model). This makes it impossible to obtain the data of the expert modules needed later (i.e., prefetching). Moreover, when multiple expert modules need to load data simultaneously, it can easily cause bandwidth peaks in High Bandwidth Memory (HBM), leading to bandwidth resource contention and slowing down the overall processing speed. Finally, the heterogeneity of compressed expert modules hinders parallel execution. After adopting N:M structured pruning and mixed precision quantization techniques, different expert modules will exhibit different computational configurations—for example, differences in module dimension, channel width, and data precision. This heterogeneity makes it difficult to achieve efficient mapping of these expert modules on a unified hardware accelerator. The originally expected parallel processing (multiple modules computing simultaneously to improve efficiency) cannot proceed smoothly, affecting the utilization efficiency of hardware resources.

[0006] It is evident that while existing technologies have solved some of the problems in deploying MoE models to a certain extent, they still have shortcomings in terms of policy adaptability, data prefetching efficiency, and hardware compatibility. They have failed to fully tap the potential of the differences in the frequency of use of MoE models by different experts, thus limiting their efficient deployment in more scenarios. Summary of the Invention

[0007] This invention provides a method and system for expert parallelism processing of large-scale language models based on MoE. This method and system can solve the problems of expert imbalance, selection delay and heterogeneity of compressed experts.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: An expert parallel processing method for large-scale language models based on MoE includes: The importance score of each expert is calculated based on the historical activation frequency, weight distribution and model structure information of the experts in the MoE model. The quantization bit width and sparsity mode of each expert are dynamically allocated according to the importance score. Based on the current hidden state, historical expert activation sequences, and importance scores of each expert, the experts to be activated are predicted; the parameters corresponding to the experts to be activated are preloaded from high-bandwidth memory into the expert cache module to generate a prefetch parameter set. Based on the expert allocation results output by the router and the list of experts to be activated, a token-expert mapping table is generated; according to the token-expert mapping table, tokens sharing the same expert are grouped, and a first-in-first-out task queue is generated for each expert to obtain the task execution queue corresponding to each expert. The computation mode and sparse decoder of the systolic array are dynamically configured based on the quantization bit width and sparse mode of each expert; tasks are scheduled according to the task execution queue to obtain the input data stream; the input data stream is transmitted to the systolic array so that the systolic array can perform mixed precision matrix multiplication and accumulation operations on the input data stream according to the computation mode and sparse decoder, and output the final inference result; wherein: during the scheduling process, if the expert parameters are hit in the prefetch parameter set, they are directly cached and loaded from the expert cache module; otherwise, the dynamic loading mechanism is triggered.

[0009] Furthermore, the importance score of each expert is calculated based on the expert's historical activation frequency, weight distribution, and model structure information in the MoE model, including: Obtain the historical activation frequency, weight distribution, and model structure information of experts in the MoE model; The importance score of each expert is calculated based on their historical activation frequency, weight distribution, and model structure information, using the following formula:

[0010] In the formula, Importance score; For expert historical activation frequency, As a weighted score, For weighted entropy, These are the weighting coefficients corresponding to the expert's historical activation frequency, weighted score, and weighted entropy, respectively. This is a temperature control function.

[0011] Furthermore, the dynamic allocation of quantization bit width and sparsity pattern for each expert based on importance scores includes: K-means clustering is used to group experts according to their importance scores. High importance clusters are assigned high quantization bit widths, and low importance clusters are assigned low quantization bit widths to obtain the quantization bit widths for each expert. Experts with importance scores greater than a preset score are classified into high importance clusters, and experts with importance scores less than or equal to a preset score are classified into low importance clusters. The sparsity is adjusted according to the expert's position in the layer; specifically, high sparsity is assigned to experts located in the middle layer of the model, and low sparsity is assigned to experts located in the input or output layer of the model, so as to obtain the sparsity pattern of each expert.

[0012] Furthermore, based on the current layer hidden state, historical expert activation sequences, and the importance scores of each expert, the experts to be activated are predicted; the parameters corresponding to the experts to be activated are preloaded from high-bandwidth memory into the expert cache module to generate a prefetch parameter set, including: The current hidden state, the historical expert activation sequence, and the importance scores of each expert are concatenated to obtain a concatenated vector. A lightweight MLP is used to predict the concatenated vector and output the experts to be activated to obtain a list of experts to be activated. The prefetching step size is dynamically optimized by maximizing the preconstructed composite utility function U, as shown in the following formula:

[0013]

[0014]

[0015] In the formula, This is a weighting factor used to adjust the prediction accuracy. With time gain The weights; m represents the prefetch step size; D KL Represents the prediction distribution of the i-th layer With the true distribution of the i-th layer The KL divergence between the two is used to measure the deviation between them; i represents the number of layer indices within the prefetch step; During the high-bandwidth memory idle phase, the parameters corresponding to the experts to be activated are preloaded from the high-bandwidth memory to the expert cache module according to the optimized prefetch step size to generate a prefetch parameter set.

[0016] Further, based on the expert allocation results output by the router and the list of experts to be activated, a token-expert mapping table is generated; according to the token-expert mapping table, tokens sharing the same expert are grouped, and a first-in-first-out task queue is generated for each expert to obtain the task execution queue corresponding to each expert, including: Obtain the expert allocation result output by the router, which is dynamically selected by the router at each layer of the MoE model based on the current hidden state of the token; The expert allocation results are combined with the list of experts to be activated to generate a token-expert mapping table, as follows: A comparator tree is used to perform a weighted fusion operation on the expert probability distributions of the expert assignment results and the expert probability distributions to output the fused expert probability distribution. Based on the fused expert probability distribution, a preset number of experts with the highest probability are selected to generate an expert list for each token; A token-expert mapping table is constructed based on the expert list and expert priority for each token. Within the same token, the expert priority is arranged in descending order of expert probability. When different tokens share the same expert, the token with higher priority is given priority to enter the task queue of this expert. Tokens sharing the same expert are grouped according to the token-expert mapping table, and a first-in-first-out task queue is built for each expert to obtain the task execution queue corresponding to each expert.

[0017] Furthermore, the computation mode and sparse decoder for dynamically configuring the systolic array based on the quantization bit width and sparse mode of each expert include: The data path width of the systolic array, the multiplier packaging strategy, and the high bandwidth memory data throughput are adjusted according to the quantization bit width of each expert. The computation mode is selected based on the data path width, multiplier encapsulation strategy, and high bandwidth memory data throughput. The computation modes include INT4 mode, FP8 mode, and BF16 mode. The grouping rules, data aggregation logic, and output alignment of the bitmask decoder are adjusted according to the sparse patterns proposed by various experts to complete the configuration of the sparse decoder.

[0018] Furthermore, based on the sparse mode activation bitmask decoder of each expert, group acquisition and splitting operations are performed on the sparse weights to align the compressed data with the smallest systolic array cell.

[0019] Further, the transmission of the input data stream to the systolic array, whereby the systolic array performs mixed-precision matrix multiplication and accumulation operations on the input data stream according to the computation mode and the sparse decoder, and outputs the final inference result, includes: Transmit the input data stream to the pulsating array; The pulsating array performs mixed-precision matrix multiplication and accumulation operations on the input data stream according to the computation mode and the sparse decoder to obtain intermediate results, and temporarily stores the intermediate results in the PSUM buffer; The post-processing module performs nonlinear operations on the intermediate results and outputs the final inference result.

[0020] An expert parallel processing system for large-scale language models based on MoE, comprising: The dynamic allocation module is used to calculate the importance score of each expert based on the historical activation frequency, weight distribution and model structure information of experts in the MoE model, and dynamically allocate the quantization bit width and sparsity mode of each expert according to the importance score. The expert prediction module is used to predict the experts to be activated based on the current hidden state, historical expert activation sequences, and importance scores of each expert; the parameters corresponding to the experts to be activated are preloaded from high-bandwidth memory to the expert cache module to generate a prefetch parameter set; The task distribution module is used to generate a token-expert mapping table based on the expert allocation results and the list of experts to be activated output by the router; according to the token-expert mapping table, the tokens sharing the same expert are grouped, and a first-in-first-out task queue is generated for each expert to obtain the task execution queue corresponding to each expert. The task execution module is used to dynamically configure the computation mode and sparse decoder of the systolic array based on the quantization bit width and sparse mode of each expert; schedule tasks according to the task execution queue to obtain the input data stream; transmit the input data stream to the systolic array so that the systolic array can perform mixed precision matrix multiplication and accumulation operations on the input data stream according to the computation mode and sparse decoder, and output the final inference result; wherein: during the scheduling process, if the expert parameters are hit in the prefetch parameter set, they are directly cached and loaded from the expert cache module; otherwise, the dynamic loading mechanism is triggered.

[0021] Furthermore, the task execution module includes a post-processing acceleration module, which is used to perform nonlinear operations on the intermediate results obtained by mixed-precision matrix multiplication and accumulation operations, and output the final inference result.

[0022] Compared with the prior art, the present invention has the following beneficial effects: This invention provides an expert parallel processing method for large-scale language models based on MoE. It achieves efficient parallel processing of the MoE model by dynamically allocating expert quantization bit width and sparsity mode, predicting and prefetching the weight parameters of experts to be calculated, generating task queues using grouped tokens, and dynamically configuring hardware accelerators. First, importance scores are calculated based on the historical activation frequency, weight distribution, and model structure of experts, thereby adaptively allocating quantization precision and sparsity ratio, avoiding resource waste and precision loss associated with a uniform strategy. Second, the hidden state of the current layer and historical activation sequences are used to predict experts to be activated, and parameters are preloaded into a dedicated cache, reducing high-bandwidth memory access and alleviating bandwidth contention. Third, tokens are grouped using a token-expert mapping table and task queues are created. Combined with the dynamic configuration of a systolic array, the heterogeneity problem of experts after compression is solved, ensuring efficient parallelism of mixed-precision matrix operations. This method improves policy adaptability, data prefetching efficiency, and hardware adaptability, unlocking the potential of the MoE model and promoting its efficient deployment and application in diverse scenarios. Attached Figure Description

[0023] Figure 1 A schematic diagram of the active prediction module of a lightweight multilayer perceptron (MLP) provided in an embodiment of the present invention; Figure 2 A schematic diagram outlining the principle architecture of the expert parallel processing method for large-scale language models based on MoE provided by this invention; wherein, (a) is a software compilation process diagram; (b) is a hardware microarchitecture diagram; and (c) is a systolic array module structure diagram. Figure 3 The diagrams provided by this invention illustrate reconfigurable arrays in three different data bit width modes; where (a) represents a diagram of 64 weights per cycle in INT4 mode; (b) represents a diagram of 32 weights per cycle in FP8 mode; and (c) represents a diagram of 16 weights per cycle in BF16 mode. Figure 4 A schematic diagram of the three-level tile mapping scheme for matrix calculation provided by the present invention; Figure 5 A schematic diagram of a mixed-precision multiplier integrated in a single DSP58 provided by the present invention; wherein, (a) is BF16×BF16; (b) is BF16×FP8(M3E4) or BF16×INT4; Figure 6 A schematic diagram illustrating the bitmask collection and splitting process of the N:M sparse mode provided by the present invention, and the alignment of the bitmask decoder output with the minimum piece PE array. Figure 7 A schematic diagram of the proactive expert management strategy consisting of two key components provided by the present invention; Figure 8A flowchart of an expert parallelism processing method for a large language model based on MoE provided by the present invention; Figure 9 This is a schematic diagram of the structure of an expert parallel processing system for a large language model based on MoE, provided by the present invention. Detailed Implementation

[0024] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0025] The following explains the technical terms used in this invention: K-means clustering is an unsupervised learning algorithm used to divide a dataset into K non-overlapping clusters, such that data points within the same cluster have high similarity, while data points in different clusters have low similarity.

[0026] ISA (Instruction Set Architecture) is the interface between computer hardware and system software. It defines the basic functional characteristics and conceptual structure of a computer and is the core part of computer architecture.

[0027] An OPU (Optical Processing Unit) is a hardware accelerator based on optical principles, designed for high-speed, low-power matrix operations (such as the core computations in machine learning).

[0028] FIFO (First In, First Out) is a logical principle for data storage and processing, widely used in computer science, electronic engineering, and industrial control.

[0029] A PE (Processing Element Array) is a computing core composed of multiple processing elements (PEs) interconnected in a specific way. It is widely used in deep learning accelerators, array processors, high-performance computing and other fields.

[0030] Lightweight Multi-Layer Perceptron (MLP) is a multilayer perceptron model optimized for resource-constrained scenarios (such as mobile devices and embedded devices). Through structural simplification, parameter compression, and improved computational efficiency, it significantly reduces model size and computational overhead while maintaining a certain level of accuracy.

[0031] Pareto optimization is a multi-objective decision-making method derived from Pareto efficiency in economics. It aims to find non-dominated solutions among multiple conflicting objectives, that is, the set of optimal solutions that cannot be improved by unilaterally improving one objective without harming the others.

[0032] DDR, short for Double Data Rate, is an advanced memory technology.

[0033] DSP is an abbreviation for Digital Signal Processor.

[0034] IP is an abbreviation for Intellectual Property, which specifically refers to reusable functional modules in chip design.

[0035] BRAM is a dedicated block random access memory inside an FPGA. It is typically distributed in fixed-capacity modules within the chip, with each module configured to have a capacity of 36Kb.

[0036] URAM is a high-density storage resource introduced in the Xilinx UltraScale architecture (an advanced FPGA architecture), with each module having a capacity of 288Kb (configured as 4K×72 bits), which is 8 times the capacity of BRAM.

[0037] As described in the background section, existing MoE model applications suffer from the following problems: First, due to imbalanced expert utilization, uniform pruning and quantization strategies are inefficient: Traditional MoE model hardware acceleration methods typically apply a fixed pruning ratio and uniform quantization precision across all experts, ignoring significant differences in expert activation frequencies. This one-size-fits-all strategy may over-prune or over-quantize highly utilized experts, reducing model accuracy, while underutilized experts retain redundant parameters, wasting computational resources. Second, expert selection latency disrupts prefetching and leads to memory bandwidth contention: In MoE inference, the active experts at each layer dynamically depend on the hidden state of the input tokens, making prefetching infeasible and triggering unpredictable HBM bandwidth spikes when multiple experts are loaded simultaneously. Third, the heterogeneity of compressed experts hinders parallel execution: After applying N:M structured space and mixed-precision quantization, experts exhibit different computational profiles—different dimensions, channel widths, and data precisions disrupt the mapping of a unified hardware accelerator.

[0038] To address the aforementioned issues, this embodiment provides an expert parallelism processing method for large-scale language models based on MoE. Through usage-aware pruning and heterogeneous compression strategies using quantization, the pruning ratio and quantization precision are dynamically adjusted based on the frequency of expert activation, reducing model size while maintaining accuracy. A lightweight adaptive expert prediction module is employed, utilizing historical patterns and hidden states to warm up the next layer of experts, achieving a hit rate of 83.4% and reducing bandwidth congestion. This system designs a reconfigurable computing architecture and proposes a systolic array with expert-aware channel reordering and parallel cascading to support efficient execution of structurally different experts.

[0039] like Figure 8 As shown, this embodiment provides an expert parallel processing method for large-scale language models based on MoE, including: The importance score of each expert is calculated based on the historical activation frequency, weight distribution and model structure information of the experts in the MoE model. The quantization bit width and sparsity mode of each expert are dynamically allocated according to the importance score. Based on the current hidden state, historical expert activation sequences, and importance scores of each expert, the experts to be activated are predicted; the parameters corresponding to the experts to be activated are preloaded from high-bandwidth memory into the expert cache module to generate a prefetch parameter set. Based on the expert allocation results output by the router and the list of experts to be activated, a token-expert mapping table is generated; according to the token-expert mapping table, tokens sharing the same expert are grouped, and a first-in-first-out task queue is generated for each expert to obtain the task execution queue corresponding to each expert. The computation mode and sparse decoder of the systolic array are dynamically configured based on the quantization bit width and sparse mode of each expert; tasks are scheduled according to the task execution queue to obtain the input data stream; the input data stream is transmitted to the systolic array so that the systolic array can perform mixed precision matrix multiplication and accumulation operations on the input data stream according to the computation mode and sparse decoder, and output the final inference result; wherein: during the scheduling process, if the expert parameters are hit in the prefetch parameter set, they are directly cached and loaded from the expert cache module; otherwise, the dynamic loading mechanism is triggered.

[0040] For example, this implementation provides an expert parallel processing method for large-scale language models based on MoE, which is designed based on the following principles, including: In this embodiment, the MoE model is effectively deployed through expert-aware adaptive quantization and sparsity. This method dynamically allocates computational resources based on the importance of each expert, optimizing the trade-off between model performance and efficiency. Key features include: First, a multi-feature importance score is employed: the core of this method is to provide an importance scoring mechanism that comprehensively considers multiple features of expert behavior. For each expert e in layer L, an importance score is calculated based on information about the expert's usage patterns and weight score distribution. The feature set includes normalized usage frequency, weight score, entropy-based metrics, and the statistical correlation between usage rate and weight score. To capture the non-uniform importance distribution among experts, this method applies a temperature-controlled Softmax transform to enhance discriminative power.

[0041] Second, Cluster-Based Quantization: This method does not use a fixed threshold, but instead uses K-means clustering to adaptively determine the quantization bit width. This method ensures that the quantization scheme adapts to the natural distribution of expert importance within the model.

[0042] Third, Position-Aware Sparsity Allocation: This proposes a position-aware sparsity allocation mechanism that takes into account the structural characteristics of the neural network. This factor modulates the sparsity mode selection, allowing intermediate layers to tolerate higher sparsity while maintaining the representational power of layers closer to the input and output. A percentile-based threshold is used to determine the final sparsity mode for each expert based on the adjusted importance score.

[0043] In this embodiment, a lightweight adaptive expert prefetching mechanism is also adopted, as follows: To achieve efficient step-size prefetching of expert parameters on resource-constrained FPGA platforms, this invention proposes an active prediction module based on a lightweight multilayer perceptron (MLP). This module comprises two key components: First, Low-Rank Constrained Lightweight MLP: The effectiveness of expert predictions in MLPs stems from the strong similarity between hidden states in the transformer layers. This method employs position-enhanced input features and a two-layer MLP structure with nonlinear activation and softmax output.

[0044] Second, a Pareto optimal dynamic step size selection (MLP Architecture) is employed: To effectively overlap prefetching latency with computation, step size prefetching advances the prediction expert requirements by m layers. However, choosing an inappropriate step size m leads to a trade-off between prediction accuracy and available time for data transmission. To address this, this method defines two span-related metrics: prediction accuracy and time gain. The former penalizes deviations from the empirically optimal step size, ensuring that the accuracy metric is sensitive to the step size; the latter measures the proportion of available time for prefetching relative to the prediction step size. An indicator function enforces a minimum step size to guarantee meaningful overlap. To select an appropriate step size, this method introduces adjustable trade-off parameters. To maximize the composite utility function.

[0045] To facilitate efficient execution of the above process, this embodiment also includes a compiler capable of mapping high-level model descriptions to low-level ISA machine code. The assembly process consists of two main flows: First, the control flow: This parses the input modeling file into a graphical intermediate representation, where each operator is described by its attributes and execution order, thus enabling operator fusion optimization. In this embodiment, the ISA is an extension of the OPU, supporting various sparse modes and mixed precision types, allowing flexible selection of computation modes within the array. The compiler schedules hardware events for each pipeline block, including data loading, PE computation, storage, and dedicated functions. During the placement phase, specific memory offsets and lengths are allocated to the data.

[0046] Second, the dataflow process involves tensor extraction of the input weight file and partitioning it across multiple cores. An appropriate tiling size is selected based on the matrix dimensions to optimize parallelism. Memory scheduling manages data movement across multiple HBM channels and DDR memory blocks. Finally, the generated instructions and weights are merged into a binary file.

[0047] To efficiently execute the aforementioned expert parallel processing method for large language models based on MoE, this embodiment also provides an expert parallel processing system for large language models based on MoE. This system aims to fully utilize high-bandwidth memory and improve the computational throughput of DSP hardware IP, specifically including: The instruction control unit module, input buffer and weight buffer module, reconfigurable systolic array module, expert management module, expert buffer module, and post-processing module are connected via a high-speed on-chip bus to achieve efficient data transmission and computation scheduling. Their specific composition and connection relationships are as follows: Command and control unit module: (1) Composition: It includes an instruction decoding unit, an instruction cache unit and timing control logic.

[0048] (2) Function: Receives instruction sequences generated by the host CPU and controls the various modules of the entire processor system to work together.

[0049] Input caching and weight caching modules: (1) Composition: The input cache and weight cache are each composed of multiple high-speed on-chip storage units.

[0050] (2) Connection: Connect to the data acquisition module respectively, providing the data and weights to be calculated.

[0051] (3) Function: Provides fast data and weight access to meet the computing needs of high bandwidth and low latency.

[0052] Reconfigurable pulsating array module: The pulse array module is composed of several cascaded processing units (PEs), and the structure of each PE is as follows: (1) The processing unit (PE) consists of an input selection multiplexer, a data input register, a control logic unit, and a calculation unit (including a matrix multiplication and accumulation unit MAC and an element-by-element calculation unit).

[0053] (2) Connection method: PE units are cascaded through three-dimensional data paths to support efficient data transmission across PEs; at the same time, each PE unit is connected to the input cache and weight cache modules; the output of the PE is connected to the partial sum cache module to store the intermediate results of each stage of calculation.

[0054] (3) Function and computation mode support: The processing unit supports different bit widths, including INT4, FP8 and BF16. Since each core is configured with 256 bits of high bandwidth memory (HBM) and 128 DSP units, and the on-chip BRAM / URAM resources are limited, the PSUM buffer can receive up to 16 FP32 results per cycle. The computation mode can be dynamically configured for different bit width data, including INT4 mode, FP8 mode and BF16 mode.

[0055] Partial and cache modules: (1) Composition: Multi-level cache register array.

[0056] (2) Connection: Connect to the output of the pulse array module and the element-by-element buffer module.

[0057] (3) Function: Store intermediate results during the calculation process, support cross-cycle accumulation of multiple rounds of calculation, and ensure efficient and stable partial sum operation.

[0058] Expert Management Module: (1) Composition: Lightweight prediction module and expert labeling packaging module.

[0059] (2) Connection: Connect to the post-processing module and ISA.

[0060] (3) Function: Lightweight prediction actively prefetches experts for the next layer, reducing memory access latency, and expert tagging and packaging for data reuse, thereby maximizing the reusability of experts and improving storage efficiency.

[0061] Expert caching module: (1) Composition: Multi-level cache register array.

[0062] (2) Connection: Connect to the weight cache module and HBM.

[0063] (3) Function: Store pre-loaded prediction experts.

[0064] Post-processing module: (4) Composition: Pipeline computing structure, including exponential function (EXP), SiLU activation function, LayerNorm normalization computing unit, etc.

[0065] (5) Connection: Connect to the partial and cache modules.

[0066] (6) Function: Efficiently completes nonlinear operations and significantly reduces the computational delay of nonlinear functions.

[0067] Based on the above modules, the overall data flow and signal transmission relationship of this system includes the following: Data input process: Data is read by the input buffer unit and passed to the systolic array module for calculation. Calculation process: After completing the matrix operation, the systolic array module outputs a portion of the result to the partial sum buffer module, and the result from the partial sum buffer module is further passed to the post-processing module for nonlinear calculation. Result output process: The processed result is transmitted back to DDR memory through a dedicated path.

[0068] like Figure 2 As shown, specifically as follows Figure 2Figures (a), (b), and (c) show the software compilation process, hardware microsystem architecture, and systolic array module for executing mixed-precision computation diagonal data flow for a large-scale language model-based expert parallel processing system based on MoE, respectively. During software compilation, the compiler optimizes the expert parallel processing method for the large-scale language model based on MoE and maps the high-level model description to low-level ISA machine code. The assembly flow consists of control flow and data flow. The hardware microsystem architecture includes an instruction control unit module, an input buffer and weight buffer module, a reconfigurable systolic array module, an expert management module, an expert buffer module, and a post-processing module. These modules are connected via a high-speed on-chip bus to achieve efficient data transmission and computation scheduling. The systolic array module is used to support different bit widths for processing units. The computation mode can be dynamically configured for different bit widths, specifically including INT4 mode, FP8 mode, and BF16 mode.

[0069] like Figure 3 As shown, specifically as follows Figure 3 Figures (a), (b), and (c) show the reconfigurable array under three different data bit width modes. It can be seen that, due to the limited on-chip BRAM / URAM resources and the configuration of 256 bits of high-bandwidth memory (HBM) and 128 DSP units per core, the PSUM buffer can receive a maximum of 16 FP32 results per cycle. For INT4 mode, 64 weights are input per cycle; for FP8 mode, 32 weights are input per cycle; and for BF16 mode, 16 weights are input per cycle.

[0070] In this embodiment, as Figure 4 As shown, the matrix computation employs a three-level tile mapping scheme. The input is [index, 2048], the weights are [2048, 1408], and the output is [index, 1408]. Leveraging the 6:8 structural sparsity and fp8 precision, the weights are compressed to [1536, 1408]. The weight matrix is ​​first partitioned along the vertical dimension to allow for greater accumulation of the PSUM buffer, followed by horizontal partitioning. To handle misalignment between the last block and the hardware array, channel reordering is performed across blocks, restoring correct accumulation behavior during computation.

[0071] In this embodiment, as Figure 5 As shown, the system or method provided in this embodiment is integrated into a mixed-precision multiplier in a DSP58. Figure 5In the diagram, Figure (a) shows BF16×BF16, where the mantissa of BF16 has only 7 bits. This reduced bit width allows two BF16 multiplications to be compressed into a single DSP58 by sharing an operand. Figure (b) shows BF16×FP8(M3E4) or BF16×INT4. Since both FP8(M3E4) and INT4 have only 3 valid mantissa bits, four multiplications can be further encapsulated into a single DSP58.

[0072] In this embodiment, as Figure 6 As shown, the bitmask collection and splitting process for the N:M sparse mode is illustrated, with the bitmask decoder output aligned with the minimum slice PE array. Assuming the PE array is constructed as a minimum 4×4 slice, the decoder output is consistently aligned with groups of four data elements. To uniformly support sparse modes from 1:4 to 8:8, the decoder input is fixed at 16 elements, and the output is limited to 4 elements. Collection refers to selectively enabling multiplexer connections based on the sparse mode. Initially, internal grouping is performed, with modes like 1:4 and 2:4 inputting in groups of 4, while 4:8 and 6:8 inputting in groups of 8. Subsequently, these grouped data are compactly aggregated to a lower exponent. Splitting refers to partitioning the data to match the array's input requirements, reserving the remaining data in registers for use in subsequent decoding steps.

[0073] The following section details the implementation of the expert parallel processing method for large-scale language models based on MoE provided in this embodiment. The specific implementation process is as follows: This embodiment provides an expert parallel processing method for large-scale language models based on MoE, including: The importance score of each expert is calculated based on the expert's historical activation frequency, weight distribution, and model structure information in the MoE model, specifically including: Obtain the historical activation frequency, weight distribution, and model structure information of experts in the MoE model; The importance score of each expert is calculated based on their historical activation frequency, weight distribution, and model structure information, using the following formula:

[0074] In the formula, Importance score; For expert historical activation frequency, As a weighted score, For weighted entropy, These are the weighting coefficients corresponding to the expert's historical activation frequency, weighted score, and weighted entropy, respectively. This is a temperature control function.

[0075] In this embodiment, a multi-dimensional expert importance scoring mechanism is introduced. A weighted formula is constructed by integrating historical activation frequency, weight distribution, and structural information, and the output is standardized using a temperature control function. This design overcomes the limitations of a single indicator, making the importance assessment more closely aligned with the actual operating characteristics of the model. This provides a scientific basis for subsequent differentiated compression strategies, ensuring a precise match between resource allocation and model accuracy requirements from the outset.

[0076] The quantization bit width and sparsity pattern of each expert are dynamically allocated based on their importance scores, specifically including: K-means clustering was used to group experts according to their importance scores. High-importance clusters were assigned high quantization bit widths, and low-importance clusters were assigned low quantization bit widths to obtain the quantization bit width for each expert. Specifically, experts with importance scores greater than a preset score were classified into high-importance clusters, and experts with importance scores less than or equal to the preset score were classified into low-importance clusters. The high quantization bit widths were 16 bits and 32 bits, and the low quantization bit widths were 4 bits, 2 bits, and 1 bit.

[0077] The sparsity is adjusted according to the expert's position in the layer; specifically, experts in the middle layer are assigned high sparsity, and experts in the input or output layer are assigned low sparsity, so as to obtain the sparsity pattern of each expert.

[0078] In this embodiment, a clustering method is used to hierarchically group experts according to their importance, and sparsity is allocated based on network location to achieve a three-dimensional customization of the compression strategy. High bit width preserves the accuracy of key experts, while low bit width compresses the resource consumption of infrequent experts; high sparsity in the middle layer meets the requirements of feature abstraction, and low sparsity in the input and output layers ensures data integrity. This strategy maximizes model performance under hardware resource constraints, achieving an optimal balance between computational accuracy and resource consumption.

[0079] Based on the current layer hidden state, historical expert activation sequences, and the importance scores of each expert, the experts to be activated are predicted. The parameters corresponding to the experts to be activated are preloaded from high-bandwidth memory into the expert cache module to generate a prefetched parameter set, specifically including: The current hidden state, the historical expert activation sequence, and the importance scores of each expert are concatenated to obtain a concatenated vector. A lightweight MLP is used to predict the concatenated vector and output the experts to be activated to obtain a list of experts to be activated. The prefetching step size is dynamically optimized by maximizing the preconstructed composite utility function U, as shown in the following formula:

[0080]

[0081]

[0082] In the formula, This is a weighting factor used to adjust the prediction accuracy. With time gain The weights; m represents the prefetch step size; D KL Represents the prediction distribution of the i-th layer With the true distribution of the i-th layer The KL divergence between the two is used to measure the deviation between them; i represents the number of layer indices within the prefetch step; During the high-bandwidth memory idle phase, the parameters corresponding to the experts to be activated are preloaded from the high-bandwidth memory to the expert cache module according to the optimized prefetch step size to generate a prefetch parameter set.

[0083] In this embodiment, a lightweight network is used to fuse real-time hidden states with historical pattern prediction expert activation, and a composite utility function is used to dynamically optimize the prefetch step size. This mechanism balances prediction accuracy and time benefits, and utilizes asynchronous preloading of parameters during idle memory periods to significantly reduce data loading latency on critical paths, effectively avoiding inference stuttering caused by bandwidth contention.

[0084] Specifically, such as Figure 1 As shown, this method comprises two key components. The first includes a low-order restricted MLP structure for hardware-friendly expert prediction: MLP-based expert prediction is effective because the hidden states between transformer layers exhibit strong similarity. It employs position-enhanced input features and a two-layer MLP structure with nonlinear activation and Softmax output. The second includes a Pareto optimization-guided dynamic step size selection mechanism to balance accuracy and latency. This embodiment defines two span-related metrics: prediction accuracy and time gain. The former penalizes deviations from the empirically optimal step size kopt, ensuring the accuracy metric is sensitive to step size. The latter measures the proportion of available prefetch time relative to the prediction step size. An indicator function enforces a minimum step size to guarantee meaningful overlap. To select an appropriate step size, this embodiment employs an adjustable tradeoff parameter λ to maximize the composite utility function.

[0085] Combination Figure 7As shown, this method includes two key components, providing an active expert management strategy comprised of two key parts. The first component is proactive expert prefetching via lightweight prediction: a lightweight prediction network is integrated into the expert management module to predict the expert IDs that may be selected for the next layer. The predictor is triggered at the beginning of each layer. During the attention computation phase, when the HBM bandwidth is not fully utilized, predicted experts are preloaded from the HBM and temporarily stored in the on-chip EP buffer. If the prediction is correct, the expert is immediately forwarded to the PE array via the Ker DMA engine for low-latency execution. In the case of a prediction error, the instruction controller dynamically inserts the expert loading operation into the execution pipeline. This mechanism is particularly effective in the decoding phase, where memory latency becomes the main performance bottleneck. The second component is expert tag packaging for data reuse: to maximize expert reusability and improve storage efficiency, target experts are used to organize the tags. This is achieved by jointly sorting the expert predictions and router outputs using a comparator tree to identify the top K experts for each token. Tokens sharing the same expert are then grouped together, and a dedicated FIFO buffer is allocated to each expert to queue the token IDs. These buffers generate token address pairs to guide the IFMDMA to obtain the correct hidden state. This design increases the reuse of expert parameters among tokens and reduces redundant memory transfers.

[0086] Based on the expert allocation results output by the router and the list of experts to be activated, a token-expert mapping table is generated. Tokens sharing the same expert are grouped according to the token-expert mapping table, and a first-in-first-out task queue is generated for each expert to obtain the task execution queue corresponding to each expert. Specifically, this includes: Obtain the expert allocation result output by the router, which is dynamically selected by the router at each layer of the MoE model based on the current hidden state of the token; The expert allocation results are combined with the list of experts to be activated to generate a token-expert mapping table, as follows: A comparator tree is used to perform a weighted fusion operation on the expert probability distributions of the expert assignment results and the expert probability distributions to output the fused expert probability distribution. Based on the fused expert probability distribution, a preset number of experts with the highest probability are selected to generate an expert list for each token; A token-expert mapping table is constructed based on the expert list and expert priority for each token. Within the same token, the expert priority is arranged in descending order of expert probability. When different tokens share the same expert, the token with higher priority is given priority to enter the task queue of this expert. Tokens sharing the same expert are grouped according to the token-expert mapping table, and a first-in-first-out task queue is built for each expert to obtain the task execution queue corresponding to each expert.

[0087] In this embodiment, a token mapping table integrating expert probabilities and a priority task queue are constructed to achieve fine-grained scheduling of computing resources. The token grouping mechanism aggregates dispersed requests, reducing redundant loading of expert tasks; the priority queue ensures high-value tasks are executed first, and the expert prefetch list synchronizes data readiness with computing demands. This design significantly improves task scheduling efficiency and enhances the continuity of parallel execution and hardware load balancing.

[0088] The computation mode and sparse decoder of the systolic array are dynamically configured based on the quantization bit width and sparse mode of each expert, specifically as follows: The data path width of the systolic array, the multiplier packaging strategy, and the high bandwidth memory data throughput are adjusted according to the quantization bit width of each expert. The computation mode is selected based on the data path width, multiplier encapsulation strategy, and high bandwidth memory data throughput. The computation modes include INT4 mode, FP8 mode, and BF16 mode. The grouping rules, data aggregation logic, and output alignment of the bitmask decoder are adjusted according to the sparse patterns of each expert to complete the configuration of the sparse decoder. Specifically, the bitmask decoder is activated according to the sparse patterns of each expert, and grouping and splitting operations are performed on the sparse weights to align the compressed data with the smallest systolic array unit.

[0089] In this embodiment, the systolic array data path and decoding rules are adaptively reconstructed based on the expert quantization bit width and sparse mode. By dynamically switching the computing mode to match different precision requirements and reconstructing the sparse data grouping alignment mechanism, the compressed heterogeneous expert modules can be directly mapped to hardware units. This technology eliminates the dependence of traditional hardware on fixed computing modes, achieving efficient and unified processing of mixed precision and sparse computing.

[0090] Tasks are scheduled according to the task execution queue to obtain the input data stream; the input data stream is transmitted to the systolic array, which performs mixed-precision matrix multiplication and accumulation operations on the input data stream according to the computation mode and sparse decoder, and outputs the final inference result, specifically including: Transmit the input data stream to the pulsating array; The pulsating array performs mixed-precision matrix multiplication and accumulation operations on the input data stream according to the computation mode and the sparse decoder to obtain intermediate results, and temporarily stores the intermediate results in the PSUM buffer; The post-processing module performs nonlinear operations on the intermediate results and outputs the final inference result.

[0091] Specifically: During the scheduling process, if the expert parameters are found in the prefetch parameter set, they are directly cached and loaded from the expert cache module; otherwise, the dynamic loading mechanism is triggered.

[0092] In this embodiment, the pulsating array performs mixed-precision multiply-accumulate operations and temporarily stores intermediate results, while the post-processing module completes the nonlinear transformation. This process avoids repeated reading and writing of intermediate data by fusing computation and post-processing operations at the hardware level, thus shortening the result output delay while ensuring computational accuracy and forming an efficient closed loop from data stream input to final inference output.

[0093] Therefore, this embodiment provides an expert parallel processing method for large-scale language models based on MoE, with the following specific advantages: First, in this method, a heterogeneous compression strategy is designed to dynamically adjust the pruning ratio and quantization accuracy according to the frequency of expert activation, thereby reducing the model size while maintaining accuracy.

[0094] Second, this method develops a compact expert prediction module that uses historical patterns and hidden states to warm up the next layer of experts, achieving a hit rate of 83.4% and reducing bandwidth congestion.

[0095] Third, this method proposes a systolic array with expert-aware channel reordering and parallel cascading to support efficient execution by structurally different experts.

[0096] like Figure 9As shown, this embodiment also provides an expert parallel processing system for a large-scale language model based on MoE, including: a dynamic allocation module, used to calculate the importance score of each expert based on the historical activation frequency, weight distribution, and model structure information of experts in the MoE model, and dynamically allocate the quantization bit width and sparsity mode of each expert according to the importance score; an expert prediction module, used to predict the experts to be activated based on the current hidden state, historical expert activation sequence, and the importance score of each expert; preload the parameters corresponding to the experts to be activated from high-bandwidth memory to the expert cache module to generate a prefetch parameter set; and a task distribution module, used to generate a token-based system based on the expert allocation result output by the router and the list of experts to be activated. An expert mapping table is used to group tokens sharing the same expert and generate a first-in-first-out task queue for each expert, resulting in a task execution queue for each expert. A task execution module is used to dynamically configure the computation mode of the systolic array and the sparse decoder based on the quantization bit width and sparsity mode of each expert. Tasks are scheduled according to the task execution queue to obtain the input data stream. The input data stream is transmitted to the systolic array, which performs mixed-precision matrix multiplication and accumulation operations on the input data stream according to the computation mode and sparse decoder, outputting the final inference result. During scheduling, if the expert parameters are found in the prefetch parameter set, they are directly cached and loaded from the expert cache module; otherwise, a dynamic loading mechanism is triggered.

[0097] The above embodiments are merely one of the implementation methods for achieving the technical solution of the present invention. The scope of protection claimed by the present invention is not limited to this embodiment, but also includes any variations, substitutions and other implementation methods that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An expert parallel processing method for large-scale language models based on MoE, characterized in that, include: The importance score of each expert is calculated based on the historical activation frequency, weight distribution and model structure information of the experts in the MoE model. The quantization bit width and sparsity mode of each expert are dynamically allocated according to the importance score. Based on the current hidden state, the historical expert activation sequence, and the importance score of each expert, the experts to be activated are predicted. The parameters corresponding to the experts to be activated are preloaded from high-bandwidth memory into the expert cache module to generate a prefetch parameter set; Based on the expert allocation results output by the router and the list of experts to be activated, a token-expert mapping table is generated. Based on the token-expert mapping table, tokens sharing the same expert are grouped, and a first-in-first-out task queue is generated for each expert to obtain the task execution queue corresponding to each expert. The computation mode and sparse decoder of the pulsating array are dynamically configured based on the quantization bit width and sparse mode of each expert. Tasks are scheduled according to the task execution queue to obtain the input data stream; The input data stream is transmitted to the systolic array, which performs mixed-precision matrix multiplication and accumulation operations on the input data stream according to the calculation mode and sparse decoder, and outputs the final inference result. During the scheduling process, if the expert parameters are found in the prefetch parameter set, they are directly cached and loaded from the expert cache module; otherwise, the dynamic loading mechanism is triggered. Based on the current layer hidden state, historical expert activation sequences, and the importance scores of each expert, the experts to be activated are predicted; the parameters corresponding to the experts to be activated are preloaded from high-bandwidth memory into the expert cache module to generate a prefetch parameter set, including: The current hidden state, the historical expert activation sequence, and the importance scores of each expert are concatenated to obtain a concatenated vector. A lightweight MLP is used to predict the concatenated vector and output the experts to be activated to obtain a list of experts to be activated. The prefetching step size is dynamically optimized by maximizing the preconstructed composite utility function U, as shown in the following formula: In the formula, This is a weighting factor used to adjust the prediction accuracy. With time gain The weights; m represents the prefetch step size; D KL Represents the prediction distribution of the i-th layer With the true distribution of the i-th layer The KL divergence between the two is used to measure the deviation between them; i represents the number of layer indices within the prefetch step; During the high-bandwidth memory idle phase, the parameters corresponding to the experts to be activated are preloaded from the high-bandwidth memory to the expert cache module according to the optimized prefetch step size to generate a prefetch parameter set; Based on the expert allocation results output by the router and the list of experts to be activated, a token-expert mapping table is generated. Tokens sharing the same expert are grouped according to the token-expert mapping table, and a first-in-first-out task queue is generated for each expert to obtain the task execution queue corresponding to each expert, including: Obtain the expert allocation result output by the router, which is dynamically selected by the router at each layer of the MoE model based on the current hidden state of the token; The expert allocation results are combined with the list of experts to be activated to generate a token-expert mapping table, as follows: A comparator tree is used to perform a weighted fusion operation on the expert probability distribution of the expert assignment results to output the fused expert probability distribution. Based on the fused expert probability distribution, a preset number of experts with the highest probability are selected to generate an expert list for each token; A token-expert mapping table is constructed based on the expert list and expert priority for each token. Within the same token, the expert priority is arranged in descending order of expert probability. When different tokens share the same expert, the token with higher priority is given priority to enter the task queue of this expert. Based on the token-expert mapping table, tokens sharing the same expert are grouped together, and a first-in-first-out task queue is built for each expert to obtain the task execution queue corresponding to each expert. The computation mode and sparse decoder for dynamically configuring the systolic array based on the quantization bit width and sparse mode of each expert include: The data path width of the systolic array, the multiplier packaging strategy, and the high bandwidth memory data throughput are adjusted according to the quantization bit width of each expert. The computation mode is selected based on the data path width, multiplier encapsulation strategy, and high bandwidth memory data throughput. The computation modes include INT4 mode, FP8 mode, and BF16 mode. The grouping rules, data aggregation logic, and output alignment of the bitmask decoder are adjusted according to the sparse patterns proposed by various experts to complete the configuration of the sparse decoder.

2. The expert parallel processing method for large-scale language models based on MoE according to claim 1, characterized in that, The importance score of each expert is calculated based on the expert's historical activation frequency, weight distribution, and model structure information in the MoE model, including: Obtain the historical activation frequency, weight distribution, and model structure information of experts in the MoE model; The importance score of each expert is calculated based on their historical activation frequency, weight distribution, and model structure information, using the following formula: In the formula, Importance score; For expert historical activation frequency, As a weighted score, For weighted entropy, These are the weighting coefficients corresponding to the expert's historical activation frequency, weighted score, and weighted entropy, respectively. This is a temperature control function.

3. The expert parallel processing method for large-scale language models based on MoE according to claim 1, characterized in that, The dynamic allocation of quantization bit width and sparsity pattern for each expert based on importance scores includes: K-means clustering is used to group experts according to their importance scores. High importance clusters are assigned high quantization bit widths, and low importance clusters are assigned low quantization bit widths to obtain the quantization bit widths for each expert. Experts with importance scores greater than a preset score are classified into high importance clusters, and experts with importance scores less than or equal to a preset score are classified into low importance clusters. The sparsity is adjusted according to the expert's position in the layer; specifically, high sparsity is assigned to experts located in the middle layer of the model, and low sparsity is assigned to experts located in the input or output layer of the model, so as to obtain the sparsity pattern of each expert.

4. The expert parallel processing method for large-scale language models based on MoE according to claim 1, characterized in that, Based on the sparse mode activation bitmask decoder of each expert, group acquisition and splitting operations are performed on the sparse weights to align the compressed data with the smallest systolic array cell.

5. The expert parallel processing method for large-scale language models based on MoE according to claim 1, characterized in that, The process of transmitting the input data stream to the systolic array, so that the systolic array can perform mixed-precision matrix multiplication and accumulation operations on the input data stream according to the computing mode and the sparse decoder, and output the final inference result, includes: Transmit the input data stream to the pulsating array; The pulsating array performs mixed-precision matrix multiplication and accumulation operations on the input data stream according to the computation mode and the sparse decoder to obtain intermediate results, and temporarily stores the intermediate results in the PSUM buffer; The post-processing module performs nonlinear operations on the intermediate results and outputs the final inference result.

6. An expert parallel processing system for large-scale language models based on MoE, used to implement the expert parallel processing method for large-scale language models based on MoE as described in any one of claims 1-5, characterized in that, include: The dynamic allocation module is used to calculate the importance score of each expert based on the historical activation frequency, weight distribution and model structure information of experts in the MoE model, and dynamically allocate the quantization bit width and sparsity mode of each expert according to the importance score. The expert prediction module is used to predict the experts to be activated based on the current hidden state, the historical expert activation sequence, and the importance score of each expert. The parameters corresponding to the experts to be activated are preloaded from high-bandwidth memory into the expert cache module to generate a prefetch parameter set; The task distribution module is used to generate a token-expert mapping table based on the expert allocation results output by the router and the list of experts to be activated. Based on the token-expert mapping table, tokens sharing the same expert are grouped, and a first-in-first-out task queue is generated for each expert to obtain the task execution queue corresponding to each expert. The task execution module is used to dynamically configure the computation mode of the systolic array and the sparse decoder based on the quantization bit width and sparse mode of each expert. Tasks are scheduled according to the task execution queue to obtain the input data stream; The input data stream is transmitted to the systolic array, which performs mixed-precision matrix multiplication and accumulation operations on the input data stream according to the calculation mode and sparse decoder, and outputs the final inference result. During the scheduling process, if the expert parameters are found in the prefetch parameter set, they are directly cached and loaded from the expert cache module; otherwise, the dynamic loading mechanism is triggered.

7. The expert parallel processing system for a large-scale language model based on MoE according to claim 6, characterized in that, The task execution module includes a post-processing acceleration module, which is used to perform nonlinear operations on the intermediate results obtained by mixed-precision matrix multiplication and accumulation, and output the final inference result.

Images