An end-side accelerator and design method of a distributed MoE large model architecture

Abstract

This invention belongs to the field of data processing and discloses an edge accelerator and its design method for a distributed MoE large model architecture. The edge accelerator includes: a distributed vector systolic array module for acquiring initial text data and performing attention calculations to obtain initial text features; a nonlinear operation module for calculating expert scores on the initial text features to obtain the target expert score corresponding to each expert in the routing network; and a MoEGate module for performing expert selection and data rearrangement processing based on the target expert scores to obtain the mapping relationship between experts and tokens, and also for performing expert inference on the initial text features based on the mapping relationship between experts and tokens to output the text inference result. This invention, by adopting a distributed integrated architecture and integrating the MoEGate module, provides a sufficient bandwidth foundation and significantly improves inference throughput and energy efficiency, greatly accelerating the processing efficiency of text data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an edge accelerator and its design method for a distributed MoE large model architecture. Background Technology

[0002] With the development of artificial intelligence, large language models have demonstrated superior performance on various tasks (such as the text processing task in this invention). To improve model capabilities and control computational costs, the Mixture of Experts (MoE) architecture has been widely adopted. The MoE model dynamically selects and activates a small subset of "expert" subnetworks for computation for each input token through a routing network, thereby greatly expanding the total parameter size of the model without significantly increasing the computational cost per inference.

[0003] However, existing MoE models lack an effective scheduling mechanism for the dynamic sparsity of MoE, and also lack the hardware infrastructure to provide sufficient memory bandwidth. MoE-LLM (Mixture of Experts-Large Language Model) models cannot be efficiently deployed on the edge, resulting in low efficiency of text processing tasks and failing to meet user needs.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide an edge-side accelerator and design method for a distributed MoE large model architecture. This aims to solve the problems in the prior art where the MoE model lacks an effective scheduling mechanism for the dynamic sparsity of MoE and also lacks a hardware infrastructure that can provide sufficient memory bandwidth. As a result, the MoE-LLM model cannot be efficiently deployed on the edge, leading to low efficiency in text processing tasks and failing to meet user needs.

[0006] To achieve the above objectives, the present invention provides an edge-side accelerator for a distributed MoE large model architecture, the edge-side accelerator for the distributed MoE large model architecture comprising: A distributed vector systolic array module is used to acquire initial text data and perform attention calculations on the initial text data to obtain initial text features; A nonlinear operation module is used to calculate expert scores on the initial text features to obtain the target expert score for each expert in the routing network. The MoEGate module is used to perform expert selection and data rearrangement processing based on the target expert score to obtain the mapping relationship between experts and tokens. It is also used to perform expert inference on the initial text features based on the mapping relationship between experts and tokens and output the text inference result.

[0007] Optionally, the edge accelerator of the distributed MoE large model architecture further includes multiple high-bandwidth memories; each of the high-bandwidth memories stores the initial text data; the distributed vector pulsation array module is composed of multiple groups of vector computing units; Each of the high-bandwidth memories is integrated in a 2.5D distributed manner with the corresponding group vector computing unit in the distributed vector systolic array module to construct an HBM channel between each high-bandwidth memory and the corresponding group vector computing unit; Each of the group vector calculation units calls the initial text data in the corresponding high-bandwidth memory through the HBM channel, and performs attention calculation on the initial text data to obtain initial text features.

[0008] Optionally, in the distributed MoE large model architecture edge accelerator, the nonlinear operation module includes a linear layer and a Softmax layer; The linear layer is used to perform linear calculations on the initial text features output by the distributed vector pulsation array module to obtain the target token; The Softmax layer is used to perform Softmax calculation on the target token using the Softmax function to obtain multiple experts corresponding to each target token in the routing network and the target expert score corresponding to each expert.

[0009] Optionally, in the distributed MoE large model architecture edge accelerator, the MoEGate module includes a direct memory access unit, a TopK extraction unit, and a real-time expert scheduler unit. The direct memory access unit is used to move the target expert score to an on-chip buffer for storage. The TopK extraction unit is used to read the target expert score in the on-chip buffer, and extract a preset number of target experts from multiple experts in descending order of the target expert score, so as to obtain a first mapping relationship between each token and the corresponding preset number of target experts; The real-time expert scheduler unit is used to perform data rearrangement processing on the first mapping relationship to obtain the mapping relationship between experts and tokens.

[0010] Optionally, in the distributed MoE large model architecture edge accelerator, the MoEGate module further includes a configuration register file unit; The configuration register file unit is used to store the mapping relationship between experts and tokens.

[0011] Optionally, in the edge accelerator of the distributed MoE large model architecture, the nonlinear operation module further includes a normalization unit. The normalization unit is used to normalize the expert scores corresponding to a preset number of target experts.

[0012] Furthermore, to achieve the above objectives, the present invention also provides a design method for an edge accelerator of a distributed MoE large model architecture, wherein the design method for the edge accelerator of the distributed MoE large model architecture includes: The distributed vector pulsation array module acquires initial text data and performs attention calculations on the initial text data to obtain initial text features; The nonlinear operation module calculates expert scores for the initial text features to obtain the target expert score for each expert in the routing network. The MoEGate module performs expert selection and data rearrangement processing based on the target expert score to obtain the mapping relationship between experts and tokens, and performs expert inference on the initial text features based on the mapping relationship between experts and tokens to output the text inference result.

[0013] Optionally, the design method of the edge accelerator in the distributed MoE large model architecture, wherein the distributed vector systolic array module acquires initial text data and performs attention calculation on the initial text data to obtain initial text features, specifically includes: Multiple high-bandwidth memories and multiple group vector computing units in the distributed vector pulsation array module are identified, and each high-bandwidth memory and its corresponding group vector computing unit are subjected to 2.5D distributed integration processing to obtain the HBM channel between each high-bandwidth memory and its corresponding group vector computing unit. Each high-bandwidth memory is determined as initial text data, and each group vector calculation unit calls the corresponding initial text data in the high-bandwidth memory through the HBM channel; Each of the group vector calculation units performs attention calculations on the initial text data to obtain initial text features.

[0014] Optionally, the design method of the edge accelerator in the distributed MoE large model architecture, wherein the nonlinear operation module calculates expert scores for the initial text features to obtain the target expert score corresponding to each expert in the routing network, specifically includes: The target token is obtained by performing linear calculations on the initial text features output by the distributed vector pulsation array module through the linear layer in the nonlinear operation module. The target token is calculated by using the Softmax function in the Softmax layer of the nonlinear operation module, thereby obtaining multiple experts corresponding to each target token in the routing network and the target expert score corresponding to each expert.

[0015] Optionally, the design method of the edge accelerator for the distributed MoE large model architecture, wherein the MoEGate module performs expert selection processing and data rearrangement processing based on the target expert score to obtain the mapping relationship between experts and tokens, and performs expert inference on the initial text features based on the mapping relationship between experts and tokens, and outputs the text inference result, specifically includes: The target expert score is moved to an on-chip buffer for storage via the direct memory access unit in the MoEGate module. The target expert score in the on-chip buffer is read by the TopK extraction unit in the MoEGate module, and a preset number of target experts are extracted from the multiple experts in descending order according to the target expert score to obtain the first mapping relationship between each token and the corresponding preset number of target experts. The first mapping relationship is rearranged by the real-time expert scheduler unit in the MoEGate module to obtain the mapping relationship between experts and tokens. Based on the mapping relationship between the experts and the tokens, expert reasoning is performed on the initial text features to output multiple initial text reasoning results; The initial text reasoning results are normalized and weighted summed to obtain the final text reasoning result.

[0016] Beneficial effects: By adopting a distributed integrated architecture and integrating the MoEGate module, this invention achieves extremely high memory bandwidth utilization and provides a sufficient bandwidth foundation. On the other hand, through intelligent scheduling of the MoEGate module, it not only effectively improves the reuse rate of expert weights, but also significantly improves inference throughput and energy efficiency, thereby greatly accelerating the processing efficiency of text data. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a preferred embodiment of the edge accelerator of the distributed MoE large model architecture of the present invention. Figure 2 This is a schematic diagram of the overall architecture of the edge accelerator of the distributed MoE large model architecture of the present invention. Figure 3 This is a schematic diagram of the execution order of the MoE layer in a preferred embodiment of the edge accelerator of the distributed MoE large model architecture of the present invention. Figure 4This is a schematic diagram of the partial bitone sorting network hardware structure of the TopK module in the MoEGate unit of the preferred embodiment of the edge accelerator of the distributed MoE large model architecture of the present invention. Figure 5 This is a schematic diagram of the hybrid expert large language model inference process in a preferred embodiment of the edge accelerator of the distributed MoE large model architecture of the present invention. Figure 6 This is a flowchart of a preferred embodiment of the design method for the edge accelerator of the distributed MoE large model architecture of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] With the development of artificial intelligence, Large Language Models (LLMs) have demonstrated outstanding performance on various tasks. To enhance model capabilities and control computational costs, the Mixture of Experts (MoE) architecture has been widely adopted. MoE models dynamically select and activate a small subset of "expert" subnetworks for each input token via a routing network, thereby significantly expanding the total parameter size of the model without substantially increasing the computational cost per inference. However, deploying MoE-LLMs on resource-constrained edge devices faces significant challenges. Existing general-purpose processors (such as CPUs and GPUs) or accelerators designed for standard dense LLMs have significant limitations when processing MoE models, including the following drawbacks: 1. Memory bandwidth bottleneck: Even if the scheduling problem is solved, the huge number of parameters in the MoE model places extremely high demands on the bandwidth of off-chip memory. Traditional integrated solutions struggle to provide sufficient HBM channels and bandwidth to meet the real-time inference requirements of large-scale MoE models.

[0020] 2. Unpredictability of Expert Weight Loading: In the MoE model, which experts will be activated for each input token is dynamically determined by the routing network at runtime. This makes it impossible to predict which expert weights need to be loaded during the compilation phase, making it difficult for traditional hardware architectures that rely on static scheduling to efficiently manage memory bandwidth and on-chip cache.

[0021] 3. Inefficient reuse of expert weights: Traditional accelerators typically process tokens sequentially, meaning they process all operations on one token before moving on to the next. In a MoE scenario, this leads to the need to repeatedly load weights from off-chip storage (such as HBM) onto the chip even if multiple consecutive tokens select the same expert, resulting in a large amount of redundant data movement and severely wasting valuable memory bandwidth and power consumption.

[0022] 4. Lack of hardware support for MoE-specific operators: The MoE model introduces key operators such as TopK to select the K highest-scoring experts from all experts. General-purpose processors are inefficient at performing such operations, and existing accelerators often lack specific optimizations for this, becoming a performance bottleneck.

[0023] In summary, existing technologies not only lack an effective scheduling mechanism for the dynamic sparsity of MoE, but also lack a hardware infrastructure that can provide sufficient memory bandwidth, which together restricts the efficient deployment of MoE-LLM on the edge.

[0024] To address the aforementioned issues, this invention proposes an edge-side inference accelerator specifically designed for Hybrid Expert Large Language Models (MoE-LLM). This accelerator employs an innovative 2.5D distributed integrated architecture and integrates the MoEGate module. This architecture achieves extremely high memory bandwidth utilization by integrating computational units (i.e., the group vector computation unit in this invention) with high-bandwidth memory (HBM) in a 2.5D distributed manner. The MoEGate module includes a TopK extraction unit based on a bitonetic ranking network and a real-time expert scheduler (RES) unit, used to perform expert selection at runtime and rearrange computational tasks from a "token-expert" format to an "expert-token" format. This hardware-software co-design provides sufficient bandwidth through the 2.5D distributed architecture and maximizes the reuse rate of expert weights through MoEGate's intelligent scheduling, thereby jointly solving the memory wall problem of MoE models during edge-side inference and significantly improving inference throughput and energy efficiency.

[0025] The edge accelerator of the distributed MoE large model architecture described in the preferred embodiment of the present invention, such as... Figure 1 , Figure 2 and Figure 3 As shown, the edge accelerator of the distributed MoE large model architecture includes a distributed vector pulsation array module, a nonlinear operation module, and a MoEGate module.

[0026] It is understood that the core of the edge-side inference accelerator of the present invention consists of three parts: a distributed group vector systolic array module (D-GVSA), a nonlinear operation module for performing nonlinear functions (such as Softmax, LayerNorm, activation functions, etc.), and a MoEGate module.

[0027] The present invention adopts a 2.5D distributed integrated architecture, which includes a distributed vector pulsating array module (D-GVSA) and a MoEGate module. The D-GVSA is composed of multiple group vector PEs (i.e., group vector computing units), each group is connected through an interconnection network and is connected to a high bandwidth memory (HBM).

[0028] Specifically, the distributed vector pulsation array module is used to acquire initial text data and perform attention calculation on the initial text data to obtain initial text features; the nonlinear operation module is used to calculate expert scores on the initial text features to obtain the target expert score corresponding to each expert in the routing network; the MoEGate module is used to perform expert selection processing and data rearrangement processing based on the target expert scores to obtain the mapping relationship between experts and tokens, and is also used to perform expert inference on the initial text features based on the mapping relationship between experts and tokens to output the text inference result.

[0029] The edge accelerator of the distributed MoE large model architecture further includes multiple high-bandwidth memories; each high-bandwidth memory stores the initial text data; the distributed vector systolic array module consists of multiple group vector computing units; each high-bandwidth memory is 2.5D distributedly integrated with the corresponding group vector computing unit in the distributed vector systolic array module to construct an HBM channel between each high-bandwidth memory and the corresponding group vector computing unit; each group vector computing unit calls the initial text data in the corresponding high-bandwidth memory through the HBM channel and performs attention calculation on the initial text data to obtain initial text features.

[0030] To address the memory bandwidth pressure caused by the large number of parameters in the MoE model, this invention adopts a 2.5D distributed integration architecture, the specific features of which are as follows: 1. Distributed design: The computing core of the edge accelerator in this invention consists of a distributed group vector systolic array (D-GVSA) composed of multiple distributed and independent memory-accessing group vector computing units (Processing Elements, PEs).

[0031] The construction process of the distributed vector pulsation array is as follows: The distributed vector systolic array has a size of Tin × Tout, where Tin is the cumulative dimension of the array and Tout is the output parallelism of the array. This array consists of Tout parallel computation units (PEs), each performing a MAC operation on the Tin features in FP16 format and the weights in INT4 format. Parallel computation is performed using Tn PEs, performing MAC operations on 1 × Tin features and Tn × Tin weights. In each cycle, only a Tin-length weight vector is loaded into the array for computation by a single PE. Meanwhile, all other vectors remain static in their respective PEs until replaced by a new vector. Feature vectors are sequentially input into the array and passed from one set of parallel vector PEs to the next. This array can output Tout FP16 data points per cycle.

[0032] 2. High-bandwidth memory integration: Multiple high-bandwidth memories (HBM) are integrated with the group vector computing units in a 2.5D distributed manner (HBM and accelerator are 2.5D integrated, and multiple ports of HBM are distributed one-to-one with the computing units (PE) of the accelerator), so that each group vector PE can access its dedicated HBM channel nearby and independently.

[0033] Effects: This distributed and proximity-based access design is one of the key advantages of this invention. It breaks the bottleneck of traditional single large computing arrays competing for a few HBM channels. Each group of vector PEs can independently and in parallel read data (including expert weights and token activation values) from its dedicated HBM channel, greatly improving the parallelism and bandwidth utilization of the overall memory subsystem.

[0034] The nonlinear operation module includes a linear layer and a Softmax layer. The linear layer is used to perform linear calculations on the initial text features output by the distributed vector pulsation array module to obtain the target token. The Softmax layer is used to perform Softmax calculations on the target token using the Softmax function to obtain multiple experts corresponding to each target token in the routing network and the target expert score corresponding to each expert.

[0035] The MoEGate module includes a direct memory access unit, a TopK extraction unit, and a real-time expert scheduler unit. The direct memory access unit is used to move the target expert score to an on-chip buffer for storage. The TopK extraction unit is used to read the target expert score from the on-chip buffer and extract a preset number of target experts from multiple experts in descending order of the target expert score to obtain a first mapping relationship between each token and the corresponding preset number of target experts. The real-time expert scheduler unit is used to perform data rearrangement processing on the first mapping relationship to obtain a mapping relationship between experts and tokens.

[0036] The MoEGate module includes a TopK extraction unit based on a bitonetic sorting network and a real-time expert scheduler (RES) for performing data format rearrangement. It selects an expert for each input token through TopK operations; uses RES to rearrange the computation task from "token-based processing" to "expert-based processing"; performs computations in expert order to reuse expert weights; and obtains expert weights by accessing their respective HBM channels in parallel through distributed GVSA units.

[0037] This invention includes a MoEGate module, which is responsible for handling the routing logic in the MoE layer and mainly includes the following modules: 1. Direct Memory Access (DMA) unit: Responsible for efficiently moving the target expert scores calculated by the routing network from the high bandwidth memory (HBM) to the on-chip buffer (the on-chip buffer is on-chip RAM, which has fast read and write speeds but small storage space, so the data of the calculation unit needs to be moved to the on-chip buffer step by step before calculation).

[0038] The formula for calculating the target expert score is as follows: ExpertScore = Softmax(W route +b); Among them, W route b and b are the weights and biases of the routing network.

[0039] 2. TopK Extraction Unit: Employs a hardware design based on a Partial Bitonic Sort Network (e.g., Figure 4 As shown in the figure, this design is highly parallelizable and pipelined, making it very suitable for hardware implementation. It can quickly select the Top-K highest scores and their corresponding expert indices from N expert scores with extremely low latency.

[0040] like Figure 4 As shown, the expert scores are sorted using a bitonic sorting network. The bitonic sorting network consists of two main steps: first, bitonic merge, which organizes the disordered input array into an array that is first monotonically increasing and then monotonically decreasing; and second, bitonic sort, which sorts the results from the first stage. Figure 4 The arrows in the diagram represent comparators, which compare the sizes of two inputs and sort the output by size. Since the hybrid expert doesn't need to sort the entire array, but only selects the larger values, only a portion of the network structure is required.

[0041] 3. Real-time Expert Scheduler (RES) Unit: The RES unit receives the output of the TopK extraction unit (i.e., the indices of the K selected experts corresponding to each token, which is also the first mapping relationship between each token and the corresponding preset number of target experts in this invention), and performs data rearrangement operation. It converts the original data stream organized by token, i.e., "Token 0 -> [Expert A, Expert C]; Token 1 -> [Expert A, Expert B]...", into a data stream organized by expert, i.e., "Expert A -> [Token 0, Token 1]; Expert B -> [Token 1]; Expert C -> [Token 0]...".

[0042] This invention provides an example of a data rearrangement operation: Assume there are 16 experts and 3 tokens in total. Each token needs to select 4 experts. The specific selection process is as follows: Token 0: Experts 0, 1, 3, 5; Token 1: Experts 0, 2, 4, 5; Token 2: Experts 1, 5, 6, 8; If executed according to the token order, the execution order is as follows: Token 0 is handled by expert 0; Token 0 is processed by expert 1; Token 0 is processed by expert 3; Token 0 is processed by expert 5; Token 1 is processed by expert 0; Proceed in sequence, the rest omitted.

[0043] If executed in the order suggested by the experts, the execution order would be: Expert 0 processes tokens 0 and 1; Expert 1 processes tokens 0 and 2; Expert 2 processes token 1; Proceed in sequence, the rest omitted.

[0044] The MoEGate module further includes a configuration register file unit; the configuration register file unit is used to store the mapping relationship between experts and tokens.

[0045] Configuration Register File (CRF): Stores the rearranged expert-token mapping relationship of the RES module, providing a scheduling basis for subsequent expert calculation units.

[0046] The nonlinear operation module further includes a normalization unit; the normalization unit is used to normalize the expert scores corresponding to a preset number of target experts.

[0047] Normalization unit: Normalizes the scores of the final selected K experts (normalization involves dividing the scores of the K selected experts by the sum of their scores) to facilitate subsequent weighted summation (weighted summation is a structural requirement of hybrid expert large language models; the structure of a hybrid expert large language model is as follows...). Figure 5 (As shown).

[0048] The reasoning process for text data in this invention is as follows: 1. Input features of the MoE layer (the input to the MoE layer is the attention score output by the attention layer (i.e., the initial text features in this invention), such as...) Figure 5 As shown, the score for each expert (i.e., the target expert score in this invention) is first generated through a linear layer and a Softmax function (which performs linear and Softmax calculations).

[0049] 2. These target expert scores are loaded onto the chip (i.e., the on-chip buffer) by MoEGate's DMA unit (Direct Memory Access Unit).

[0050] 3. The Top-K extraction unit selects the Top-K experts corresponding to each token (the Top-K extraction unit is used to read the target expert scores in the on-chip buffer and extract a preset number of target experts from multiple experts in descending order of the target expert scores). 4. The RES module reads the TopK results (the first mapping relationship between each token and the corresponding preset number of target experts) and performs a format conversion from "token-expert" to "expert-token" (that is, to obtain the mapping relationship between experts and tokens).

[0051] 5. The converted scheduling information is written into the CRF (i.e., the configuration register file unit in this invention).

[0052] 6. In the subsequent expert calculation phase, the calculation unit will no longer execute according to the token order, but according to the expert order. For each activated expert, the calculation unit will process all the tokens assigned to that expert at once.

[0053] Technical effects of the present invention: 1. Hardware and software synergy to completely solve the memory wall: MoEGate's RES scheduling strategy maximizes the temporal locality of expert weights and reduces unnecessary repeated loading; while the 2.5D distributed integration architecture maximizes memory bandwidth utilization, ensuring that the system has sufficient bandwidth to support multiple experts when they need to perform parallel calculations.

[0054] 2. Significantly improve the reuse rate of expert weights: By rearranging the data in the RES module, this invention fundamentally solves the problem of repeated loading of weights in the traditional token-based processing mode.

[0055] 3. Improve memory bandwidth utilization: The distributed design of D-GVSA ensures high-bandwidth connections between computing units (i.e., group vector computing units) and HBM (i.e., high-bandwidth memory), avoiding bandwidth contention and enabling parallel computing tasks scheduled by MoEGate to truly obtain the required memory bandwidth.

[0056] 4. Reduce inference latency and power consumption: Reducing the number of memory accesses and making full use of high bandwidth directly leads to improved inference speed and reduced power consumption.

[0057] The key innovation of this invention lies in: 1. Co-design of 2.5D distributed integrated architecture and MoEGate: The 2.5D distributed computing architecture, which provides a high bandwidth foundation, is combined with the MoEGate module to solve the dynamic sparsity scheduling problem.

[0058] 2. The overall architecture of the MoEGate hardware operation unit, especially the integration of the TopK extraction unit and the real-time expert scheduler (RES).

[0059] 3. The dynamic data rearrangement method from "token-expert" to "expert-token" executed by the RES module.

[0060] Possible design changes or modifications to this invention are as follows: 1. Alternatives to TopK extraction units: Although bitone networks are the preferred solution, other efficient hardware sorting algorithms can also be used.

[0061] 2. Implementation of the RES module: In addition to using dedicated logic, the RES function can also be implemented by using a small, high-bandwidth on-chip SRAM in conjunction with address mapping logic.

[0062] 3. Scheduling granularity: The present invention currently relies entirely on expert rearrangement, but more complex scheduling strategies can be considered in the future.

[0063] Furthermore, such as Figure 6 As shown, based on the aforementioned distributed MoE large model architecture edge accelerator, this invention also provides a design method for the distributed MoE large model architecture edge accelerator, wherein the design method for the distributed MoE large model architecture edge accelerator includes: Step S10: The distributed vector pulsation array module acquires initial text data and performs attention calculation on the initial text data to obtain initial text features.

[0064] Step S20: The nonlinear operation module calculates expert scores for the initial text features to obtain the target expert score for each expert in the routing network.

[0065] Step S30: The MoEGate module performs expert selection and data rearrangement processing based on the target expert score to obtain the mapping relationship between experts and tokens, and performs expert reasoning on the initial text features based on the mapping relationship between experts and tokens to output the text reasoning result.

[0066] Specifically, in step S10, the distributed vector pulsation array module acquires initial text data and performs attention calculations on the initial text data to obtain initial text features, which specifically includes the following steps: Multiple high-bandwidth memories and multiple group vector computing units in the distributed vector pulsation array module are identified, and each high-bandwidth memory and its corresponding group vector computing unit are subjected to 2.5D distributed integration processing to obtain the HBM channel between each high-bandwidth memory and its corresponding group vector computing unit. Each high-bandwidth memory is determined as initial text data, and each group vector calculation unit calls the corresponding initial text data in the high-bandwidth memory through the HBM channel; Each of the group vector calculation units performs attention calculations on the initial text data to obtain initial text features.

[0067] Specifically, in step S20, the nonlinear operation module calculates expert scores for the initial text features to obtain the target expert score for each expert in the routing network, which includes the following steps: The target token is obtained by performing linear calculations on the initial text features output by the distributed vector pulsation array module through the linear layer in the nonlinear operation module. The target token is calculated by using the Softmax function in the Softmax layer of the nonlinear operation module, thereby obtaining multiple experts corresponding to each target token in the routing network and the target expert score corresponding to each expert.

[0068] Specifically, in step S30, the MoEGate module performs expert selection and data rearrangement processing based on the target expert score to obtain the mapping relationship between experts and tokens, and performs expert inference on the initial text features based on the mapping relationship between experts and tokens, outputting the text inference result. This specifically includes the following steps: The target expert score is moved to an on-chip buffer for storage via the direct memory access unit in the MoEGate module. The target expert score in the on-chip buffer is read by the TopK extraction unit in the MoEGate module, and a preset number of target experts are extracted from the multiple experts in descending order according to the target expert score to obtain the first mapping relationship between each token and the corresponding preset number of target experts. The first mapping relationship is rearranged by the real-time expert scheduler unit in the MoEGate module to obtain the mapping relationship between experts and tokens. Based on the mapping relationship between the experts and the tokens, expert reasoning is performed on the initial text features to output multiple initial text reasoning results; The initial text reasoning results are normalized and weighted summed to obtain the final text reasoning result.

[0069] The beneficial effects of this invention are as follows: By adopting a distributed integrated architecture and integrating the MoEGate module, this invention achieves extremely high memory bandwidth utilization and provides a sufficient bandwidth foundation. On the other hand, through intelligent scheduling of the MoEGate module, it not only effectively improves the reuse rate of expert weights, but also significantly improves inference throughput and energy efficiency, thereby greatly accelerating the processing efficiency of text data.

[0070] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. An edge-side accelerator for a distributed MoE large model architecture, characterized in that, The edge accelerators of the distributed MoE large model architecture include: A distributed vector pulsation array module, which is composed of multiple groups of vector calculation units; Multiple high-bandwidth memories, each of which stores initial text data; Each of the high-bandwidth memories is integrated in a 2.5D distributed manner with the corresponding group vector computing unit in the distributed vector systolic array module to construct an HBM channel between each high-bandwidth memory and the corresponding group vector computing unit; Each of the group vector calculation units calls the initial text data in the corresponding high-bandwidth memory through the HBM channel, and performs attention calculation on the initial text data to obtain initial text features; A nonlinear operation module is used to calculate expert scores on the initial text features to obtain the target expert score for each expert in the routing network. The MoEGate module is used to perform expert selection and data rearrangement processing based on the target expert score, obtain the mapping relationship between experts and tokens, and perform expert inference on the initial text features based on the mapping relationship between experts and tokens, and output the text inference result.

2. The edge accelerator of the distributed MoE large model architecture according to claim 1, characterized in that, The nonlinear operation module includes a linear layer and a Softmax layer; The linear layer is used to perform linear calculations on the initial text features output by the distributed vector pulsation array module to obtain the target token; The Softmax layer is used to perform Softmax calculation on the target token using the Softmax function to obtain multiple experts corresponding to each target token in the routing network and the target expert score corresponding to each expert.

3. The edge accelerator of the distributed MoE large model architecture according to claim 2, characterized in that, The MoEGate module includes a direct memory access unit, a TopK extraction unit, and a real-time expert scheduler unit. The direct memory access unit is used to move the target expert score to an on-chip buffer for storage. The TopK extraction unit is used to read the target expert score in the on-chip buffer, and extract a preset number of target experts from multiple experts in descending order of the target expert score, so as to obtain a first mapping relationship between each token and the corresponding preset number of target experts; The real-time expert scheduler unit is used to perform data rearrangement processing on the first mapping relationship to obtain the mapping relationship between experts and tokens.

4. The edge accelerator of the distributed MoE large model architecture according to claim 1, characterized in that, The MoEGate module also includes a configuration register file unit; The configuration register file unit is used to store the mapping relationship between experts and tokens.

5. The edge accelerator of the distributed MoE large model architecture according to claim 3, characterized in that, The nonlinear operation module also includes a normalization unit; The normalization unit is used to normalize the expert scores corresponding to a preset number of target experts.

6. A design method for an edge-side accelerator based on the distributed MoE large model architecture described in any one of claims 1-5, characterized in that, The design method for the edge accelerator of the distributed MoE large model architecture includes: The distributed vector pulsation array module acquires initial text data and performs attention calculations on the initial text data to obtain initial text features; The nonlinear operation module calculates expert scores for the initial text features to obtain the target expert score for each expert in the routing network. The MoEGate module performs expert selection and data rearrangement processing based on the target expert score to obtain the mapping relationship between experts and tokens, and performs expert inference on the initial text features based on the mapping relationship between experts and tokens to output the text inference result.

7. The design method of the edge accelerator for the distributed MoE large model architecture according to claim 6, characterized in that, The distributed vector systolic array module acquires initial text data and performs attention calculations on the initial text data to obtain initial text features, specifically including: Multiple high-bandwidth memories and multiple group vector computing units in the distributed vector pulsation array module are identified, and each high-bandwidth memory and its corresponding group vector computing unit are subjected to 2.5D distributed integration processing to obtain the HBM channel between each high-bandwidth memory and its corresponding group vector computing unit. Each high-bandwidth memory is determined as initial text data, and each group vector calculation unit calls the corresponding initial text data in the high-bandwidth memory through the HBM channel; Each of the group vector calculation units performs attention calculations on the initial text data to obtain initial text features.

8. The design method of the edge accelerator for the distributed MoE large model architecture according to claim 6, characterized in that, The nonlinear computation module calculates expert scores for the initial text features to obtain the target expert score for each expert in the routing network, specifically including: The target token is obtained by performing linear calculations on the initial text features output by the distributed vector pulsation array module through the linear layer in the nonlinear operation module. The target token is calculated by using the Softmax function in the Softmax layer of the nonlinear operation module, thereby obtaining multiple experts corresponding to each target token in the routing network and the target expert score corresponding to each expert.

9. The design method of the edge accelerator for the distributed MoE large model architecture according to claim 8, characterized in that, The MoEGate module performs expert selection and data rearrangement based on the target expert score to obtain a mapping relationship between experts and tokens. It then performs expert inference on the initial text features based on this mapping relationship, outputting the text inference result, specifically including: The target expert score is moved to an on-chip buffer for storage via the direct memory access unit in the MoEGate module. The target expert score in the on-chip buffer is read by the TopK extraction unit in the MoEGate module, and a preset number of target experts are extracted from the multiple experts in descending order according to the target expert score to obtain the first mapping relationship between each token and the corresponding preset number of target experts. The first mapping relationship is rearranged by the real-time expert scheduler unit in the MoEGate module to obtain the mapping relationship between experts and tokens. Based on the mapping relationship between the experts and the tokens, expert reasoning is performed on the initial text features to output multiple initial text reasoning results; The initial text reasoning results are normalized and weighted summed to obtain the final text reasoning result.

10. The design method of the edge accelerator for the distributed MoE large model architecture according to claim 9, characterized in that, The process of rearranging the first mapping relationship through the real-time expert scheduler unit in the MoEGate module to obtain the mapping relationship between experts and tokens further includes: The mapping relationship between experts and tokens is stored in the configuration register file unit in the MoEGate module. When expert reasoning is required, the mapping relationship between experts and tokens is retrieved from the configuration register file unit.

Images