Token rearrangement control apparatus and method for hybrid expert model

Abstract

This invention relates to a token reordering control device based on a hybrid expert model, comprising: a routing mask parsing unit for real-time capture of expert distribution target information and parsing to generate an expert target bitmap; a dynamic index offset generator for calculating local offsets and base addresses within the target expert slices and generating a packing write address corresponding to each token; a multi-expert arbitrator for arbitrating concurrent packing requests from multiple experts; a bridging matrix for switching data paths; and a packing read / write data path for completing zero-copy packing during the data loading and data write-back phases using logical address remapping. This invention also relates to a token reordering control method based on a hybrid expert model. This invention enables real-time hardware capture and parsing of expert distribution targets, zero-copy compact packing of non-contiguous token features, fair arbitration and conflict resolution during concurrent packing by multiple experts, and adaptive splitting scheduling and pipeline bubble elimination when the lengths of expert slices differ significantly.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically to a token rearrangement control device and method using a hybrid expert model. Background Technology

[0002] In recent years, large-scale language models based on Transformers have made groundbreaking progress in fields such as natural language processing, computer vision, and multimodal understanding. From OpenAI's GPT series and Google's Gemini to Meta's Llama series and China's DeepSeek series, the scale of model parameters has expanded from billions to hundreds of billions or even trillions. However, the explosive growth in model size has also led to a sharp increase in training and inference costs, making it one of the core challenges in the field of artificial intelligence to achieve larger model capacity with limited computing power.

[0003] Mixture-of-Experts (MoE) models have emerged as a promising model scaling technique. The core idea is to replace the single feedforward network layer in the traditional Transformer with multiple parallel expert sub-networks, while introducing a lightweight gating network, the router, responsible for dynamically and sparsely selecting a small subset of the most suitable experts for computation for each input token. This conditional computation mechanism allows for a significant expansion of the model's total number of parameters, as only a subset is activated in each inference iteration. This enables a substantial increase in model capacity and performance while maintaining a relatively constant computational cost per inference.

[0004] However, the MoE model suffers from unbalanced routing load and idle low-activation-rate experts. Overloaded popular experts cause their computation queues to accumulate a large number of unprocessed tokens, becoming a latency bottleneck for the entire MoE layer, while the computational resources of other experts are idle, resulting in a decrease in overall hardware utilization. Low-activation-rate experts are not activated for a long time, and their stored knowledge cannot be effectively utilized, resulting in wasted computational resources and potentially reducing the quality of model output.

[0005] To address the aforementioned issues, the following technical solutions have been proposed: a software-based Scatter-Gather scheme, which implements token rearrangement and distribution at the software level; a DMA controller-based Scatter-Gather scheme, which utilizes the existing DMA engine on the chip to perform non-contiguous memory access operations; an acceleration scheme based on general-purpose sparse computing hardware, which uses existing sparse computing hardware to accelerate sparse operations in MoE; a "space-for-time" scheme based on large-capacity on-chip storage and high-bandwidth interconnects, which alleviates data rearrangement pressure by providing massive on-chip SRAM and high-bandwidth interconnect networks; a software optimization scheme based on prefetching and speculation, which pre-arranges tokens to the target expert's local memory by speculating the expert routing path of the token, and then uses a lightweight Transformer predictor to predict the routing paths of all layers at once, realizing proactive expert prefetching and cross-batch token rearrangement; and a routing load balancing scheme based on software random numbers, which adds software-generated random noise to the Logit value output by the router, and then redistributes tokens through complex floating-point Softmax probability calculations.

[0006] While existing technical solutions have certain advantages, they still have the following significant drawbacks: 1) Unavoidable software scatter-gather overhead and CPU backoff: Expert sorting and grouping, as well as token-by-token indexing and transport, are all handled by software or the CPU. Control latency increases with the number of tokens, becoming a significant component of the end-to-end latency of the MoE operator. CPU backoff at the entire layer when the operator shape does not support it further amplifies the latency; 2) Waste of non-contiguous access bandwidth: Non-contiguous token features require multiple small-granularity reads and writes. If transferred through a temporary rearrangement buffer, the data undergoes a two-stage "read-write-read" process, significantly reducing the effective bandwidth utilization of on-chip SRAM and generating a large amount of ineffective dynamic power consumption; 3) Pipeline bubbles: Routing, grouping, and computation are executed serially, resulting in PE... The array idles during software grouping; and the number of tokens selected by different experts is dynamically uneven, the general matrix multiplication size of each expert is very different, and the sequential or batch execution of each expert further aggravates the bubble, resulting in low array utilization; 4) Limited scalability: When the number of tokens and experts increases, the software grouping overhead increases approximately linearly, which is difficult to match the continuous improvement of NPU computing throughput. The MoE operator is difficult to leverage the computing power benefits brought by sparse activation on the edge NPU. Summary of the Invention

[0007] The purpose of this invention is to provide a token rearrangement control device and method based on a hybrid expert model, so as to eliminate software scatter-gather overhead and CPU rollback, improve on-chip effective bandwidth utilization, eliminate hardware pipeline bubbles, and improve scalability.

[0008] This invention provides a token rearrangement control device based on a hybrid expert model, comprising: The routing mask parsing unit is configured to capture expert distribution target information in real time and parse it to generate an expert target bitmap; The dynamic index offset generator is configured to calculate the local offset and base address within the target expert slice based on the expert target bitmap, and generate the packing write address corresponding to each Token. A multi-expert arbitrator, configured to arbitrate concurrent packaged requests from multiple experts; The bridging matrix is ​​configured to switch data paths based on the arbitration results of the multi-expert arbitrator; The data path for packaging and reading is configured to use logical address remapping to complete zero-copy packaging during the data loading and data write-back phases.

[0009] Furthermore, the routing mask parsing unit includes: The expert index decoder is configured to stream route records from the route log buffer unit in token order, obtain the token sequence number, k expert indices, and gating weight from the route records, and transmit the k expert indices to the E-bit decoding network. The E-bit decoding network decodes each expert index to generate a one-hot vector; where 1≤k≤K max K max The maximum number of distribution experts per token supported by the hardware; The bitmap assembly logic unit is configured to receive k one-hot vectors corresponding to the same Token and merge the k one-hot vectors into an E-bit expert target bitmap of the Token. The mask SRAM is configured to store the expert target bitmap in token order.

[0010] Furthermore, the mask SRAM is organized in a multi-bank block structure, with the number of banks equal to the number of parallel read ports of the dynamic index offset generator. Each bank reads out M expert target bitmaps of tokens in each cycle, where M is determined by the data path width and the bit width of the expert target bitmap.

[0011] Furthermore, the routing mask parsing unit employs streaming control logic and uses a valid-ready streaming handshake and backpressure mechanism with the router.

[0012] Furthermore, the dynamic index offset generator includes: The parallel prefix sum tree calculation unit is configured to receive E columns of expert target bitmaps, perform parallel prefix sum tree calculation on the E columns of expert target bitmaps, and obtain the prefix sum final value of each column of expert target bitmaps; the prefix sum final value obtained by parallel prefix sum tree calculation of each column of expert target bitmaps represents the slice length of the expert slice; the prefix sum of each column represents the local offset of the corresponding token; The expert count accumulator is configured to perform cumulative calculations according to the slice length of all expert slices in expert order, to obtain the base address of each expert slice in the expert slice area of ​​the global buffer unit, and store it in the base address table; The write address assembly logic unit is configured to receive the local offset of the Token output by the parallel prefix sum tree calculation unit and the base address in the base address table, and generate a packaged write address for each Token.

[0013] Furthermore, the parallel prefix sum tree calculation unit adopts Brent-Kung parallel prefix sum tree calculation, with a tree depth of O(log T), where T is the number of tokens in the batch.

[0014] Furthermore, the write address assembly logic unit is also configured to output a write enable signal and a byte mask for bit-by-bit compression writing by the packing write port.

[0015] Furthermore, the multi-expert arbitrator includes: The weighted round-robin arbitration unit is configured to count the tokens corresponding to the expert slices with the weight values. Anti-hunger counter, configured to limit the maximum number of wait times for expert slices; The Bank Conflict Resolver is configured to schedule multiple packet write requests out of turn when they access the same SRAM Bank in the same period.

[0016] Furthermore, the packaged read / write data path is configured as follows: during the loading phase, based on the packaged write address generated by the dynamic index offset generator, the non-contiguously distributed token features in the global buffer unit are concatenated to the continuous expert slice area; during the write-back phase, based on the mapping relationship retained during packaging, the expert output is inversely mapped back to the original token order position, completing the end-to-end zero-copy process.

[0017] The present invention also provides a token rearrangement control method based on a hybrid expert model, comprising: Step S1: Based on the above token rearrangement control device, during the forward computation of the MoE layer, the router obtains the token sequence number expert index and gating weight, and generates routing logs which are stored in the routing log cache unit. Step S2: The routing mask parsing unit stream reads the routing log, decodes the expert index of each Token into an E-bit expert target bitmap, and writes it compactly into the mask SRAM in Token order. Step S3: The dynamic index offset generator performs a parallel prefix sum on each column of expert target bitmap to obtain the local offset of each token within its target expert slice. At the same time, the expert counter accumulator counts the length of each slice and generates the base address corresponding to each expert slice. Step S4: The multi-expert arbitrator arbitrates the concurrent packet write requests from multiple experts. The bridging matrix, based on the base address and local offset obtained in step S3, periodically pipelines the discontinuously distributed token features in the global buffer unit to the continuous expert slice area during the data loading stage. Step S5: The processing unit array sequentially reads each expert slice and performs dense GEMM. Step S6: The expert output is reverse mapped back to the original token order position according to the mapping relationship retained during packaging, completing the end-to-end zero-copy process.

[0018] This invention embeds a hardware streaming token rearrangement control device between the global buffer unit and the processing unit array. The hardware captures the expert distribution target output by the router in real time and automatically assembles discontinuous token features into compact, continuous expert slices in a periodic pipeline manner, achieving dynamic Scatter-Gather with zero CPU intervention. This invention enables real-time hardware capture and parsing of expert distribution targets, zero-copy compact packaging of discontinuous token features, fair arbitration and conflict resolution during concurrent packaging by multiple experts, and adaptive splitting scheduling and pipeline bubble elimination when uneven expert load leads to large differences in the length of expert slices. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the layout and structure of the token rearrangement control device based on the hybrid expert model according to Embodiment 1 of the present invention.

[0020] Figure 2 for Figure 1 A schematic diagram of the structure of the middle routing mask parsing unit.

[0021] Figure 3 for Figure 1 A schematic diagram of the structure of the dynamic index offset generator.

[0022] Figure 4 Arbitration strategy and data pathway diagram for multi-expert concurrent processing.

[0023] Figure 5 This is a flowchart of a token rearrangement control method based on a hybrid expert model according to Embodiment 2 of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0025] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0026] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0027] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use. These terms are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] Furthermore, terms such as "horizontal," "vertical," and "sag" do not imply that components must be absolutely horizontal or suspended, but rather that they can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal relative to "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0029] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0030] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0031] Example 1: Token Reordering Control Device Based on Hybrid Expert Model like Figure 1 As shown, the token rearrangement control device 100 of the hybrid expert model provided in this embodiment is set on the data path between the global buffer unit 200 and the processing unit array 300 and connected to the router 400. It includes a route mask parsing unit 110, a dynamic index offset generator 120, a multi-expert arbitrator 130, a bridging matrix 140, and a packaged read / write data path.

[0032] The routing mask parsing unit 110 is connected to the routing log caching unit 410. The routing log caching unit 410 stores expert distribution target information, and the routing mask parsing unit 110 captures the expert distribution target information in real time and parses it to generate an expert target bitmap. The dynamic index offset generator 120 reads the expert target bitmap and generates a packaged write address corresponding to each token. The multi-expert arbitrator 130 receives concurrent packaging requests from multiple experts and performs arbitration; the bridging matrix 140 switches the packaged read / write data path. The packaged read / write data path completes zero-copy packaging in the data loading and write-back stages using logical address remapping. One end of the path is connected to the read / write port of the global buffer unit 200, and the other end is connected to the loading / write-back channel of the processing unit array 300 via the multi-expert arbitrator 130 and the bridging matrix 140.

[0033] The token rearrangement control device 100 performs actual data transfer using logical address remapping. It operates in two phases: data loading and data write-back. In the loading phase, according to the packing write address given by the dynamic index offset generator 120, the non-contiguous token features in the global buffer unit 200 are automatically assembled into a compact and continuous expert slice area in a periodic pipeline. The entire process is achieved through logical address remapping, without generating intermediate data copies. In the write-back phase, the expert output is reverse-mapped back to the original token order position according to the mapping relationship retained during packing, achieving zero copy across the entire link.

[0034] like Figure 2 As shown, the routing mask parsing unit 110 includes an expert index decoder 111, a bitmap assembly logic unit 112, and a mask SRAM (random static memory) 113 connected in sequence. The index decoder 111, the bitmap assembly logic unit 112, and the mask SRAM 113 all adopt streaming control logic.

[0035] The expert index decoder 111 is configured to stream route records from the route log buffer unit 410 in token order, retrieve the token sequence number, k expert indices, and gating weights from the route records, transmit the k expert indices to the E-bit decoding network, and transmit the token sequence number and gating weights to the bypass timing alignment register. The E-bit decoding network decodes each expert index to generate a one-hot vector, with a decoding delay of one cycle. The expert index decoder 111 streams and synchronously outputs the k one-hot vectors, the token sequence number, and the gating weights.

[0036] Bitmap assembly logic unit 112 is configured to receive k one-hot vectors corresponding to the same Token, and merge these k one-hot vectors bitwise ORed into an E-bit expert target bitmap for that Token. The value of k satisfies 1 ≤ k ≤ K. max K max The maximum number of distribution experts per token supported by the hardware, when working in Top... 1 mode, k equals 1, one The hot vector itself is the expert target bitmap.

[0037] The masked SRAM 113 is configured to store the expert target bitmap in a compact manner according to the token order. The read and write addresses are automatically generated by the streaming control logic without software intervention. The masked SRAM 113 adopts a multi-bank block organization. The number of banks is equal to the number of parallel read ports of the dynamic index offset generator 120. Each bank can read M tokens of the expert target bitmap in each cycle, where M is determined by the data path width and the bit width of the expert target bitmap.

[0038] The routing mask parsing unit 110 globally adopts streaming control logic and uses a valid-ready streaming handshake and backpressure mechanism with the router 400. The routing log caching unit 410 acts as a resilient FIFO to decouple the rate fluctuations between the router 400 and the rearrangement control device 100. When the mask SRAM 113 is full, a backpressure signal is generated to pause parsing, ensuring that no tokens are lost.

[0039] In MoE inference, after a batch of tokens is routed, each token is assigned to an expert or one of the top-k experts. Token features need to be written to the expert slice area in global buffer unit 200, meaning each expert has an independent slice to store all token features routed to that expert. Since the token order is not consistent with expert affiliation, writing directly in token order would result in discontinuous data within the expert slices, making it difficult to read sequentially during subsequent expert computation.

[0040] The function of the dynamic index offset generator 120 is to calculate the local offset of each token within the target expert slice and combine it with the expert slice base address to generate the final write address; it also generates write enable and byte mask to support bit-compressed writing of the packaged write port. The entire process requires low latency and is best completed in a single cycle or several cycles, therefore a parallel prefix sum tree is used.

[0041] like Figure 3 As shown, the dynamic index offset generator 120 is implemented based on parallel prefix sums, and includes a parallel prefix sum tree calculation unit 121, an expert count accumulator 122, and a write address assembly logic unit 123.

[0042] The local offset of a token within its target expert slice is equal to the number of selected tokens preceding it in that column of the expert target bitmap, i.e., the prefix sum of the binary sequence. In this embodiment, the parallel prefix sum tree calculation unit 121 is connected to the aforementioned mask SRAM 113 and is configured to receive the E-column expert target bitmap output by the mask SRAM 113. It performs parallel Brent-Kung parallel prefix tree calculations on the E-column expert target bitmaps, with a tree depth of O(log T), where T is the number of tokens in the batch, to obtain the final value of the prefix sum of each column of the expert target bitmap. The Brent-Kung parallel prefix tree first performs forward reduction calculation of the interval sum using a binary reduction tree, and then performs fan-out propagation completion using a reverse tree. This allows for the generation of the local offset of the entire batch of tokens in a single cycle or several cycles, exhibiting the advantage of low hardware area overhead. The final value of the prefix sum obtained by the parallel prefix sum tree calculation for each column of the expert target bitmap represents the token count of that expert slice, i.e., the slice length of that expert slice; the prefix sum of each column is the local offset offset_i of that token.

[0043] The expert count accumulator 122 is configured to perform cumulative calculations according to the slice length of all expert slices and in expert order to obtain the base address base_i of each expert slice in the expert slice area of ​​the global buffer unit 200, and store it in the base address table. In this embodiment, the base address table is implemented by a small-capacity register file.

[0044] The write address assembly logic unit 123 is configured to receive the token local offset output by the parallel prefix sum tree calculation unit 121 and the base address stored in the base address table, and generate a packet write address for each token: addr(j) ​​= base_e(j) + offset_e(j) (j) × feature granularity In the formula, e(j) is the target expert of the j-th token, and the feature granularity is the number of SRAM units occupied by the feature vector of a single token.

[0045] In addition to outputting the packing write address addr(j), the write address assembly logic unit 123 also synchronously outputs the write enable signal and byte mask, which are used by the packing write port to compress the data bit by bit. This allows the non-contiguous token features to be automatically spliced ​​into a compact expert slice at a single-cycle granularity during data loading.

[0046] When the number of tokens in a batch is large, a segmented pipeline structure of intra-segment prefix sum + inter-segment accumulation can be adopted to make a trade-off between hardware area and number of computation cycles. At this time, the dynamic index offset generator 120 is also reused as the reverse mapping address generation unit in the write-back stage.

[0047] The multi-expert arbitrator 130 receives concurrent packet requests from multiple experts and performs arbitration. The bridging matrix 140 switches the packet read / write data path. Multi-expert concurrent processing is implemented through a three-layer mechanism: "read-side shared broadcast + write-side arbitration and packetization + conflict resolution and load balancing," as follows: Figure 4 As shown, the read-side shared broadcast network is used to read the feature vector of the same token from the global buffer unit 200 only once when the same token is selected by multiple experts, and send it to multiple expert computing channels simultaneously through the broadcast network to avoid repeated reading of bandwidth on the large chip. When the packaged write requests of E expert slices compete for limited write ports, the multi-expert arbitrator 130 adopts weighted round-robin arbitration, with the weight taken from the token count of the expert slice (long slices obtain continuous multi-slot port occupancy), and sets an anti-starvation counter to limit the maximum waiting number of small slices to ensure fairness and scheduling determinism. A bank conflict resolver is also set up to schedule the wrong slots when multiple packaged write requests access the same SRAM Bank in the same period. At the same time, the base address of each expert slice is allocated with staggered offset according to the number of banks to reduce the probability of bank conflicts from the address layout. The load balancing and slice splitting module is used to deal with the dynamic differences in the length of each expert slice caused by the unbalanced routing load. When the length of a single expert slice exceeds the preset threshold, it is split into multiple sub-slices and mapped to multiple computing channels in parallel for execution. The splitting information is recorded in the slice descriptor to improve the utilization of the processing unit array 300. The multi-expert arbitrator 130 outputs a unified scheduling table (including expert number, token number, read / write address, and channel number) per cycle, driving the bridging matrix 140 and the packaged read / write data path to ensure the seamless connection between packaged read / write and the computation of the processing unit array 300.

[0048] Example 2: Token Reordering Control Method Based on Hybrid Expert Model like Figure 5 As shown in the figure, the token rearrangement control method based on a hybrid expert model provided in this embodiment includes the following steps: Step S1: Based on the aforementioned token rearrangement control device, during the forward computation of the MoE layer, router 400 obtains expert distribution target information and stores it in the routing log cache unit 410. Specifically, router 400, executed by the NPU vector unit, calculates the gating score token by token, outputs the Top-k expert indices and gating weights, and writes them to the routing log.

[0049] In step S2, the routing mask parsing unit 110 stream reads the routing log, decodes the expert index of each Token into an E-bit expert target bitmap, and writes it compactly into the mask SRAM 113 in Token order. Here, the i-th bit being 1 indicates that the Token is assigned to expert i; the i-th bit being 0 indicates that the Token is not assigned to expert i, and E is the total number of experts.

[0050] In step S3, the dynamic index offset generator 120 performs a parallel prefix sum on each column of expert target bitmaps to obtain the local offset of each token within its target expert slice. Simultaneously, the expert counter accumulator 122 counts the length of each slice and generates the base address corresponding to each expert slice. The output consists of a mapping table in the form {Token ID, expert index, local offset} and a base address table in the form {expert index, base address}. Each expert corresponds to one column of expert target bitmaps, and the base address represents the starting address of each expert slice in the global buffer unit 200.

[0051] In step S4, the multi-expert arbitrator 130 arbitrates the concurrent packet write requests from multiple experts. The bridging matrix 140 automatically assembles the non-contiguously distributed token features in the global buffer unit 200 into a compact and continuous expert slice area in a periodic pipeline during the data loading stage, based on the base address and local offset. Zero-copy packaging is achieved throughout the process using logical address remapping, without generating intermediate data copies. The read and write addresses of the packet read and write data paths are derived from the mapping table and base address table of the dynamic index offset generator 120, and read / write enable is controlled by the scheduling table output by the multi-expert arbitrator 130.

[0052] Step S5: The processing unit array 300 sequentially reads each expert slice and performs dense GEMM. The slice addresses are continuous and support full burst pipeline reading.

[0053] Step S6: The expert output is reverse mapped back to the original token order position according to the mapping relationship retained during packaging, or directly output in a compact form according to the layout requirements of the downstream operator, achieving zero copying across the entire chain.

[0054] Steps S2 to S4 above form the i-th batch of token rearrangement pipeline, and step S5 forms the (i-1)-th batch of computation pipeline. While the rearrangement control device processes the i-th batch of tokens, the processing unit array 300 computes the (i-1)-th batch of packaged expert slices. The routing, rearrangement, and computation stages overlap, eliminating the serial waiting time of "computation can only be performed after grouping is complete" in existing technologies. Therefore, the expert slice area is organized using a double-buffered (ping-pong) system: when the i-th batch of slices is written to buffer A, the processing unit array 300 reads the (i-1)-th batch of slices from buffer B; the two buffers are used alternately, and the writing of slices and the reading of computations do not block each other.

[0055] It should be noted that when the number of tokens in a batch is 0 (empty batch): the prefix sum of each column is 0, there is no write request for packaging, and the pipeline skips this batch directly; when the length of an expert slice is 0 (no tokens are distributed to this expert): the base address table entry of this expert exists but there is no write request, the processing unit array 300 skips the expert's GEMM, without generating additional overhead other than bubbles; routing log buffer overflow risk: the mask SRAM is full → parsing is paused → routing log cache accumulates → FIFO is about to be full → the hierarchical backpressure chain of backpressure to the router ensures that no tokens are lost, and the FIFO depth is designed according to the rate ratio of the router and the rearrangement control device; when the slice length exceeds the splitting threshold: split into sub-slices, the length of the sub-slices does not exceed the threshold, and ensure that the load of each computing channel is relatively balanced.

[0056] The above-mentioned token rearrangement control device and method based on the hybrid expert model have the following beneficial effects: (1) Completely eliminate software scatter-gather overhead and CPU backoff in MoE routing scheduling. Expert distribution target parsing, index offset calculation and slice packaging are all completed by dedicated hardware streaming, and overlap with routing calculation and expert GEMM calculation pipelines. The control latency is reduced from the software level (microsecond to millisecond level) to the hardware cycle level; there is no need for temporary buffer rearrangement and secondary handling, which significantly reduces on-chip invalid memory access and dynamic power consumption; no longer relying on CPU backoff path, the MoE operator can be executed in full-link hardware on the NPU, which greatly reduces control latency and on-chip invalid power consumption.

[0057] (2) It realizes compact packing and efficient parallel reading of non-contiguous tokens in on-chip SRAM, which significantly improves the utilization of on-chip effective bandwidth. After packing, the expert slice addresses are continuous, and the PE Array is read in full burst mode, eliminating the bandwidth loss caused by small-granularity random memory access; zero-copy packing avoids intermediate data copies, and the saved bandwidth is directly converted into effective computing bandwidth.

[0058] (3) Eliminate hardware pipeline bubbles and significantly improve the overall computing throughput of the NPU when running large model MoE operators. The routing-reordering-computation pipeline overlaps, and the PE Array no longer idles while waiting for software grouping; the multi-expert concurrent arbitration, slice splitting and weighted round-robin mechanism can adapt to the dynamic differences in slice length of each expert caused by the unbalanced routing load, minimize the impact of load imbalance on the pipeline, and the GEMM of each expert is executed continuously in pipeline. The overall computing throughput of the MoE operator is significantly improved and approaches the peak of hardware computing.

[0059] (4) Low intrusion and easy to promote. The rearranged controller reuses the existing on-chip memory and DMA system, with small hardware increments, and only affects the MoE layer operators, without affecting dense model inference; it is compatible with the existing NPU compiler toolchain, which is convenient for promotion and application on various NPUs (including edge and end-side).

[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. Various variations can be made to the above embodiments of the present invention. That is, all simple and equivalent changes and modifications made based on the claims and description of this invention fall within the protection scope of the claims of this patent. All aspects not described in detail in this invention are conventional technical content.

Claims

1. A token rearrangement control device based on a hybrid expert model, characterized in that, Located on the data path between the global buffer unit and the processing unit array, and connected to the router, it includes: The routing mask parsing unit is configured to capture expert distribution target information in real time and parse it to generate an expert target bitmap; The dynamic index offset generator is configured to calculate the local offset and base address within the target expert slice based on the expert target bitmap, and generate the packing write address corresponding to each Token. A multi-expert arbitrator, configured to arbitrate concurrent packaged requests from multiple experts; The bridging matrix is ​​configured to switch data paths based on the arbitration results of the multi-expert arbitrator; The data path for packaging and reading is configured to use logical address remapping to complete zero-copy packaging during the data loading and data write-back phases.

2. The token rearrangement control device for a hybrid expert model according to claim 1, characterized in that, The routing mask parsing unit includes: The expert index decoder is configured to stream route records from the route log buffer unit in token order, obtain the token sequence number, k expert indices, and gating weight from the route records, and transmit the k expert indices to the E-bit decoding network. The E-bit decoding network decodes each expert index to generate a one-hot vector; where 1≤k≤K max K max The maximum number of distribution experts per token supported by the hardware; The bitmap assembly logic unit is configured to receive k one-hot vectors corresponding to the same Token and merge the k one-hot vectors into an E-bit expert target bitmap of the Token. The mask SRAM is configured to store the expert target bitmap in token order.

3. The token rearrangement control device for a hybrid expert model according to claim 2, characterized in that, The mask SRAM is organized in a multi-bank block structure. The number of banks is equal to the number of parallel read ports of the dynamic index offset generator. Each bank reads out M expert target bitmaps of tokens in each cycle, where M is determined by the data path width and the bit width of the expert target bitmap.

4. The token rearrangement control device for a hybrid expert model according to claim 2, characterized in that, The routing mask parsing unit uses streaming control logic and employs a valid-ready streaming handshake and backpressure mechanism with the router.

5. The token rearrangement control device for a hybrid expert model according to claim 1, characterized in that, The dynamic index offset generator includes: The parallel prefix sum tree calculation unit is configured to receive E columns of expert target bitmaps, perform parallel prefix sum tree calculation on the E columns of expert target bitmaps, and obtain the prefix sum final value of each column of expert target bitmaps; the prefix sum final value obtained by parallel prefix sum tree calculation of each column of expert target bitmaps represents the slice length of the expert slice; the prefix sum of each column represents the local offset of the corresponding token; The expert count accumulator is configured to perform cumulative calculations according to the slice length of all expert slices in expert order, to obtain the base address of each expert slice in the expert slice area of ​​the global buffer unit, and store it in the base address table; The write address assembly logic unit is configured to receive the local offset of the Token output by the parallel prefix sum tree calculation unit and the base address in the base address table, and generate a packaged write address for each Token.

6. The token rearrangement control device for a hybrid expert model according to claim 5, characterized in that, The parallel prefix sum tree computation unit adopts Brent-Kung parallel prefix sum tree computation with a tree depth of O(log T), where T is the number of tokens in the batch.

7. The token rearrangement control device for a hybrid expert model according to claim 5, characterized in that, The write address assembly logic unit is also configured to output a write enable signal and a byte mask for bit-by-bit compression writing by the packing write port.

8. The token rearrangement control device for a hybrid expert model according to claim 1, characterized in that, The multi-expert arbitrator includes: The weighted round-robin arbitration unit is configured to count the tokens corresponding to the expert slices with the weight values. Anti-hunger counter, configured to limit the maximum number of wait times for expert slices; The Bank Conflict Resolver is configured to schedule multiple packet write requests out of turn when they access the same SRAM Bank in the same period.

9. The token rearrangement control device for a hybrid expert model according to claim 1, characterized in that, The packaged read / write data path is configured as follows: during the loading phase, based on the packaged write address generated by the dynamic index offset generator, the non-contiguously distributed token features in the global buffer unit are concatenated to the continuous expert slice area; during the write-back phase, based on the mapping relationship retained during packaging, the expert output is inversely mapped back to the original token order position, completing the end-to-end zero-copy process.

10. A token rearrangement control method using a hybrid expert model, characterized in that, include: Step S1: Based on the token rearrangement control device according to any one of claims 1 to 9, during the forward computation of the MoE layer, the router obtains the token sequence number expert index and gating weight, and generates routing logs which are stored in the routing log cache unit. Step S2: The routing mask parsing unit stream reads the routing log, decodes the expert index of each Token into an E-bit expert target bitmap, and writes it compactly into the mask SRAM in Token order. Step S3: The dynamic index offset generator performs a parallel prefix sum on each column of expert target bitmap to obtain the local offset of each token within its target expert slice. At the same time, the expert counter accumulator counts the length of each slice and generates the base address corresponding to each expert slice. Step S4: The multi-expert arbitrator arbitrates the concurrent packet write requests from multiple experts. The bridging matrix, based on the base address and local offset obtained in step S3, periodically pipelines the discontinuously distributed token features in the global buffer unit to the continuous expert slice area during the data loading stage. Step S5: The processing unit array sequentially reads each expert slice and performs dense GEMM. Step S6: The expert output is reverse mapped back to the original token order position according to the mapping relationship retained during packaging, completing the end-to-end zero-copy process.

Images