Large language model indexing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses an indexing method and device for a large language model, computer equipment and a storage medium, and the method comprises the steps: generating an expert mask table based on a comparison result of a generalized coding sequence outputted by a router and a preset numerical value; executing a first type of prefix sum operation on the expert mask table to generate a first auxiliary index table, and integrating tail row data of the first auxiliary index table under a plurality of batch processing scenes to generate a second auxiliary index table; executing a second type of prefix sum operation on the tail row of the second auxiliary index table to generate a third auxiliary index table; according to the first auxiliary index table, the second auxiliary index table and the third auxiliary index table, rearranging input feature data to obtain rearranged feature data; performing expert-level batch general matrix multiplication on the rearranged feature data; and based on the gating weight output by the router, performing weighted fusion operation on results of the batch general matrix multiplication operation to obtain a final output feature of a big language model MOE architecture reasoning stage.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of artificial intelligence, and in particular to an indexing method, apparatus, computer device, and storage medium for a large language model. Background Technology

[0002] As the scale of Large Language Models (LLMs) continues to expand, achieving efficient inference with limited hardware resources has become a key challenge. The Mixture of Experts (MoE) architecture, by introducing a "sparse activation" mechanism, has become one of the mainstream technical approaches to address this challenge. The core idea of ​​the MoE model is to decompose the massive model parameters into multiple relatively small "expert" networks, and then use a gating network (router) to dynamically select and activate a few of the most relevant experts for each input token, thereby significantly reducing the forward computation overhead of a single input while maintaining the enormous scale of the total model parameters.

[0003] Traditional MoE inference implementations, especially on AI accelerators such as Neural Processing Units (NPUs), typically employ a token-by-token General Matrix-Vector Multiplication (GEMV) model. Specifically, for each token in the sequence, the system sequentially reads the weight parameters of the corresponding expert from memory based on its routing result (i.e., generalized one-hot encoding), performs a GEMV calculation once, and finally weights and fuses the results of K experts. This method is intuitive and easy to implement.

[0004] However, token-by-token GEMV operations are inherently serial or low-parallel, failing to fully utilize the massively parallel computing capabilities of modern NPUs / GPUs, especially the high-throughput general-purpose matrix multiplication units. When processing long sequences or high-concurrency (multi-batch) requests, a large number of fine-grained GEMV operations can lead to insufficient utilization of computational cores, becoming a bottleneck for inference performance. Moreover, to support dynamic routing, the system typically needs to pre-allocate sufficient memory space for all experts to accommodate intermediate computation results in the worst-case scenario. This requires the allocated space to be the maximum sequence length × hidden layer dimension × total number of experts. For large-scale MoE models with a large number of experts, this pre-allocation strategy results in huge, and in most cases, underutilized, memory overhead, limiting the scale of model deployment and batch size. Summary of the Invention

[0005] In view of this, embodiments of this application provide at least one indexing method, apparatus, computer device, and storage medium for a large language model.

[0006] The technical solution of this application embodiment is implemented as follows: On one hand, embodiments of this application provide an indexing method for a large language model, the method comprising: An expert mask table is generated based on the comparison results between the generalized coded sequence output by the router and the preset values. Perform a first type of prefix sum operation on the expert mask table to generate a first auxiliary index table, and integrate the last row data of the first auxiliary index table in multiple batch processing scenarios to generate a second auxiliary index table; Perform a second type of prefix sum operation on the last row of the second auxiliary index table to generate a third auxiliary index table; Based on the first auxiliary index table, the second auxiliary index table and the third auxiliary index table, the input feature data is rearranged to obtain rearranged feature data; Perform expert-level batch general matrix multiplication on the rearranged feature data; Based on the gating weights output by the router, a weighted fusion operation is performed on the results of the batch general matrix multiplication operation to obtain the final output features of the MOE architecture inference stage.

[0007] On the other hand, embodiments of this application provide another indexing device for large language models, the method comprising: The processing module is used to generate an expert mask table based on the comparison results between the generalized coded sequence output by the router and the preset values. Perform a first type of prefix sum operation on the expert mask table to generate a first auxiliary index table, and integrate the last row data of the first auxiliary index table in multiple batch processing scenarios to generate a second auxiliary index table; Perform a second type of prefix sum operation on the last row of the second auxiliary index table to generate a third auxiliary index table; Based on the first auxiliary index table, the second auxiliary index table and the third auxiliary index table, the input feature data is rearranged to obtain rearranged feature data; Perform expert-level batch general matrix multiplication on the rearranged feature data; The output module is used to perform a weighted fusion operation on the results of the batch general matrix multiplication operation based on the gating weights output by the router, so as to obtain the final output features of the inference stage of the large language model MOE architecture.

[0008] In another aspect, embodiments of this application provide a computer device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described indexing method for a large language model.

[0009] In another aspect, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described indexing method for a large language model.

[0010] In another aspect, embodiments of this application provide a computer program including computer-readable code, wherein when the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps in the indexing method for implementing the above-described large language model.

[0011] In another aspect, embodiments of this application provide a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described indexing method for a large language model.

[0012] This application's embodiments systematically transform the traditional, low-parallelism per-token general matrix-vector multiplication inference mode into a higher-parallelism per-expert batch general matrix multiplication mode by generating and utilizing a series of index auxiliary tables. This significantly improves the computational efficiency of the NPU at the MoE layer, especially in high-concurrency inference scenarios. Simultaneously, this mechanism optimizes the data layout in memory, eliminating the need to reserve maximum possible space for each expert when pre-allocating storage space; instead, it only needs to allocate space for the actually selected experts, thereby drastically reducing memory usage. The entire solution primarily relies on basic comparison and addition operations, has no special requirements for hardware circuitry, is highly versatile, and is easy to implement and deploy on existing NPU architectures.

[0013] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0015] Figure 1 A schematic diagram illustrating the implementation process of an indexing method for a large language model provided in this application embodiment; Figure 2 A schematic diagram of the principle of a prior art provided for an embodiment of this application; Figure 3 One of the schematic diagrams illustrating the principle of an indexing method for a large language model provided in this application embodiment; Figure 4A second schematic diagram illustrating the principle of an indexing method for a large language model provided in this application embodiment; Figure 5 The third schematic diagram illustrating the principle of an indexing method for a large language model provided in this application embodiment; Figure 6 The fourth schematic diagram illustrating the principle of an indexing method for a large language model provided in this application embodiment; Figure 7 Fifth schematic diagram illustrating the principle of an indexing method for a large language model provided in this application embodiment; Figure 8 A schematic diagram (sixth in the series) illustrating the principle of an indexing method for a large language model provided in this application embodiment; Figure 9 The seventh schematic diagram illustrating the principle of an indexing method for a large language model provided in this application embodiment; Figure 10 The eighth schematic diagram illustrating the principle of an indexing method for a large language model provided in this application embodiment; Figure 11 A schematic diagram illustrating the composition of an indexing device for a large language model, provided in an embodiment of this application; Figure 12 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0018] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0020] MoE (Mixture of Experts) is the core structure of large models that improves parameter scale and efficiency through "sparse activation". Its core is "multi-expert division of labor + gated routing".

[0021] Reference Figure 2 The MoE consists of three parts: 1) multiple "expert networks" (such as small Transformer blocks, responsible for calculating specific features); 2) a gating network (routing module, which selects a few experts to participate in the calculation based on the features of the input token); and 3) a result fusion module (which sums the outputs of the selected experts according to their weights to obtain the final result). Its formula can be expressed as follows: (1) (1) Where Top-K(G(x)) represents the K expert indices with the largest weights in G(x), x is the input token feature, and y is the output feature after passing through the MoE layer. This represents the weight assigned to the i-th expert by the gating network. This represents the output of the i-th expert network to input x.

[0022] One-hot encoding in MoE is a sparse vector that "records the token-expert mapping relationship," and it has two core forms: Standard one-hot encoding (K=1 hard routing): The vector length equals the total number of experts. Only the selected expert position is 1, and the rest are 0. It is used for "single expert activation" scenarios, such as the routing record of a simplified version of MoE. For example: the total number of experts is 16, and expert 5 is selected. Then the corresponding standard one-hot encoding is: [0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0].

[0023] Generalized one-hot encoding (K>1 soft routing): The vector has only the top-K selected expert positions with non-zero values ​​(values ​​are routing weights, such as softmax probabilities), and the rest are 0. It marks both "which experts were selected" and records "the contribution ratio of each expert", and is the core mapping carrier of mainstream MoE. For example: if the total number of experts is 16, and 4 experts are selected, and the selected experts are 1, 5, 10 and 15, then the corresponding generalized one-hot encoding is: [0.1 0 0 0 0.5 0 0 0 0 0.2 0 0 0 00.2 0].

[0024] This invention proposes an efficient NPU indexing mechanism for large-scale MoE model inference. Compared to the token-by-token GEMV inference mode, it significantly improves computational efficiency, especially in high-concurrency large-scale model inference scenarios. It effectively solves the multi-expert space pre-allocation problem, significantly reducing the allocated space. Furthermore, this mechanism requires no additional hardware circuitry. This invention effectively overcomes the pain points of low computational parallelism, large pre-allocation space, and the need for additional complex hardware functions in large-scale MoE model inference.

[0025] This application provides an indexing method for a large language model, which can be executed by a processor of a computer device. The computer device can refer to a server, laptop, tablet, desktop computer, smart TV, set-top box, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or other similar computer equipment. Figure 1 This is a schematic diagram illustrating the implementation process of an indexing method for a large language model provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes: Step 101: Generate an expert mask table based on the comparison result between the generalized coding sequence output by the router and the preset value.

[0026] In this embodiment, the router refers to the gated network module of the Hybrid Experts (MoE) layer in the large language model. Its function is to calculate and output a generalized coding sequence based on the input features. This sequence indicates which expert networks each input token should be assigned to and their corresponding weights. The generalized coding sequence is a sparse vector representation whose length is equal to the total number of experts. Only a small number of experts (Top-K) have non-zero values ​​(i.e., routing weights) at their corresponding positions, while the remaining positions are zero. The preset value is usually zero, used to compare with each element in the generalized one-hot coding to determine whether the expert corresponding to that position is activated. The expert mask table is a two-dimensional matrix or tensor, whose rows typically correspond to tokens in the input sequence, and whose columns correspond to all experts. Each element in the table is a binary value (0 or 1) used to record whether each token has selected the corresponding expert.

[0027] The system receives a generalized encoded sequence output by the router module. The sequence has dimensions of (batch size, sequence length, total number of experts), where each element represents the routing weight of the corresponding token and expert. The system compares each element in the sequence with a preset value (e.g., zero). This comparison generates a binary mask matrix with the same dimensions. In this generated mask matrix, if the value at the corresponding position in the original generalized encoded sequence is greater than the preset value (i.e., the expert is selected), the corresponding position in the mask matrix is ​​set to 1; otherwise, it is set to 0. Finally, the system stores this generated binary mask matrix as an expert mask table, which clearly records the activation relationship between each token and each expert throughout the entire input batch.

[0028] Step 102: Perform a first type of prefix sum operation on the expert mask table to generate a first auxiliary index table, and integrate the last row data of the first auxiliary index table in multiple batch processing scenarios to generate a second auxiliary index table.

[0029] In this embodiment, the first type of prefix sum operation is an algorithmic operation that accumulates and sums data along a specific dimension (usually along the token sequence dimension, i.e., the row direction). For a sequence, the prefix sum at position i is the sum of all elements in the sequence from the start position to position i. The first auxiliary index table is an intermediate data table obtained by performing the first type of prefix sum operation on the expert mask table. Its structure is usually consistent with the dimensions of the expert mask table, but the value at each position represents the cumulative number of times the corresponding expert has been activated from the beginning of the sequence to the current position. The last row of data refers to the last row of data corresponding to each independent batch data block in the first auxiliary index table. In multiple batch processing scenarios, the system processes the data of each batch separately. The second auxiliary index table is another intermediate table generated by integrating the last row data of the first auxiliary index tables of multiple batches. Its rows usually correspond to different batch processing indices, and its columns correspond to experts. The values ​​in the table represent the total number of times each expert is activated in each independent batch (i.e., the total number of tokens that selected that expert in that batch).

[0030] The system reads the generated expert mask table. It then performs a first-type prefix sum operation along the row direction (i.e., the token sequence dimension) of the expert mask table. Specifically, for each column (corresponding to one expert), starting from the first row, the system adds the value of the current row to the cumulative sum of the previous row, using the result as the new value for the current row, and recursively calculates this for all subsequent rows. After the operation is complete, a first auxiliary index table is generated. In scenarios where multiple batches are processed simultaneously, the system extracts the corresponding last row data from the first auxiliary index table for each independent batch data block. These last row data represent the cumulative total number of tokens activated for each expert within each batch. The system integrates these extracted last row data according to the batch processing order, arranging them into a new table to generate a second auxiliary index table.

[0031] Step 103: Perform a second type of prefix sum operation on the last row of the second auxiliary index table to generate a third auxiliary index table.

[0032] In this embodiment, the second type of prefix sum operation is another cumulative summation operation, but its operation dimension differs from that of the first type of prefix sum operation. In this step, it is performed on the second auxiliary index table along the batch dimension (typically the row direction). The third auxiliary index table is one of the final key index tables obtained by performing the second type of prefix sum operation on the second auxiliary index table. Each position in this table stores an offset whose value represents the total cumulative token count of all experts before being assigned to a particular expert globally (across all processed batches), or, from another perspective, the prefix sum of the total cumulative token count of a particular expert across all batches up to the current batch.

[0033] The system obtains the generated second auxiliary index table. The system then performs a second type of prefix sum operation on this second auxiliary index table along the row direction (i.e., the batch processing dimension). Specifically, for each column in the table (corresponding to one expert), starting from the row corresponding to the first batch, the system adds the value of the current row to the cumulative sum of the previous row, using the result as the new value for the current row, and recursively calculates this for all rows. After completing this prefix sum operation, the system obtains a third auxiliary index table. This table provides global quantity offset information at the expert level, which is crucial for locating the starting address of each expert's data during subsequent data rearrangement.

[0034] Step 104: Based on the first auxiliary index table, the second auxiliary index table, and the third auxiliary index table, rearrange the input feature data to obtain rearranged feature data.

[0035] In this embodiment, input feature data refers to the raw terminal input data that needs to be processed by the MoE layer, and its dimensions are typically (batch size, sequence length, hidden layer dimension). Reordering refers to the process of reorganizing the elements in the input feature data from their original token-ordered organization to an expert-ordered organization according to specific indexing rules. The reordered feature data refers to the new data layout obtained after the reordering operation, where the organization method becomes: first, grouping by expert index, and then within each expert group, arranging them according to the order in which tokens are assigned to that expert.

[0036] The system performs a rearrangement operation on the input raw feature data based on the index information contained in the first, second, and third auxiliary index tables. First, the system uses the third auxiliary index table to determine the starting offset address of the data corresponding to each expert in the rearranged buffer. Next, for each token, the system uses the second auxiliary index table to determine the local offset of its batch block under that expert. Then, the system queries the first auxiliary index table (or, if hardware supports it, its optimized variant, such as the token-corresponding expert order table) to obtain the precise order position of the token within its selected expert. Combining these offsets and order information, the system calculates the target storage address of the token's feature vector in the rearranged data buffer. The system copies or moves the feature data of each token from its original position in the input data to the calculated target address. Finally, all feature data is reorganized to form rearranged feature data stored contiguously according to expert order.

[0037] Step 105: Perform expert-level batch general matrix multiplication on the rearranged feature data.

[0038] In this embodiment, expert-level batch general matrix multiplication refers to the operation mode of treating the feature data of all tokens to be processed as a whole matrix for each expert network in the hybrid expert model, and performing a one-time matrix multiplication calculation with the parameter matrix of that expert network. General matrix multiplication is a basic linear algebra operation that calculates the product of two matrices.

[0039] The system iterates through all experts from 1 to N. For the i-th expert currently being processed, the system retrieves information related to that expert from the third auxiliary index table to determine the total number of tokens that need to be processed by that expert and locates the starting address of its input data in the rearranged feature data buffer. The system extracts the feature vectors of all tokens assigned to expert i from the rearranged feature data and combines them into a two-dimensional matrix. Simultaneously, the system loads the parameter matrix corresponding to expert i. The system calls the matrix multiplication unit to perform a batch general matrix multiplication operation on the extracted feature matrix and the expert parameter matrix. This operation completes the calculation of all tokens assigned to expert i in one go, generating the output matrix of expert i. The system repeats the above process for each expert, sequentially completing the batch matrix multiplication calculation for all experts.

[0040] Step 106: Based on the gating weights output by the router, perform a weighted fusion operation on the results of the batch general matrix multiplication operation to obtain the final output features of the MOE architecture inference stage.

[0041] In this embodiment, the gating weights output by the router refer to the specific values ​​at the non-zero positions in the generalized coding sequence generated by the router in step 101. These values ​​represent the contribution weight of each selected expert to its corresponding token. The weighted fusion operation is a linear combination operation, that is, for each token, the output vectors of all its selected experts are multiplied by the corresponding gating weights, and then the weighted vectors are summed. The final output feature of the large language model inference layer refers to the output tensor obtained after weighted fusion by all selected experts in the MoE layer. This tensor will be used as the input of subsequent network layers.

[0042] The system obtains the generalized encoded sequence containing specific weights (or the Top-K weight table extracted from it) initially output by the router. For each token in the sequence, the system locates the specific position of the token in the output matrix of each expert, calculated in step 105, based on its corresponding expert selection index, and extracts the corresponding output vector. The system performs scalar multiplication on the extracted output vectors from K different experts with the corresponding gating weights assigned to the token by the router. The system then adds the K weighted vectors element-wise to obtain a fused output vector. The system performs the above operations on all tokens, ultimately generating and outputting the final feature tensor of the entire batch after processing by the MoE layer.

[0043] This application's embodiments systematically transform the traditional, low-parallelism per-token general matrix-vector multiplication inference mode into a higher-parallelism per-expert batch general matrix multiplication mode by generating and utilizing a series of index auxiliary tables. This significantly improves the computational efficiency of the NPU at the MoE layer, especially in high-concurrency inference scenarios. Simultaneously, this mechanism optimizes the data layout in memory, eliminating the need to reserve maximum possible space for each expert when pre-allocating storage space; instead, it only needs to allocate space for the actually selected experts, thereby drastically reducing memory usage. The entire solution primarily relies on basic comparison and addition operations, has no special requirements for hardware circuitry, is highly versatile, and is easy to implement and deploy on existing NPU architectures.

[0044] In some embodiments, step 101 includes: Step 1011: Compare each element value in the generalized encoded sequence with zero.

[0045] In this embodiment, the generalized coded sequence is a data structure whose length is equal to the total number of expert networks. Each element value in the sequence represents the routing weight of the corresponding expert network selected by the current input token. A non-zero element value indicates that the expert is selected and its value represents the contribution weight, while a zero value indicates that the expert is not selected. The element value is a specific numerical value stored in the sequence. Zero is a baseline numerical constant used for comparison.

[0046] The system acquires a generalized coded sequence generated by a gating network. The system iterates through each element in the sequence. For each element, the system performs a comparison operation, comparing the element value with the value zero. The purpose of this comparison is to determine whether the element value is greater than zero, thereby determining whether its corresponding expert network has been selected by the current input token.

[0047] Step 1012: When the element value is greater than zero, the comparison result is marked as a first value. When the element value is equal to or less than zero, the comparison result is marked as a second value.

[0048] In this embodiment, the first value is a predefined flag value used to represent a logical true or false state, such as the integer 1. The second value is a predefined flag value used to represent a logical false or no state, such as the integer 0. The comparison result is the logical judgment result generated after comparing the element value with zero in step 1011.

[0049] The system assigns a label based on the comparison result. When the comparison result indicates that the element value is greater than zero, the system generates and records the first value. When the comparison result indicates that the element value is equal to or less than zero, the system generates and records the second value. This process generates a binary logical label for the selection relationship between each input token and each expert network.

[0050] Step 1013: Arrange the first value and the second value according to the original dimensions of the generalized coding sequence to form a two-dimensional matrix containing the first value and the second value. The two-dimensional matrix is ​​the expert mask table. The row dimension of the expert mask table corresponds to the length of the input token sequence, and the column dimension corresponds to the total number of expert networks. The expert mask table is used to indicate the selection relationship between each input token and each expert network. The first value represents that the corresponding expert is selected by the current token, and the second value represents that the corresponding expert is not selected by the current token.

[0051] In this embodiment, the original dimension refers to the inherent shape of the generalized encoded sequence, specifically its length, which is equal to the total number of expert networks. A two-dimensional matrix is ​​a data structure with both row and column dimensions. The expert mask table is the final generated two-dimensional table used to record the relationship between all tokens and all expert choices. The row dimension corresponds to the length of the input token sequence, i.e., the total number of tokens in the batch. The column dimension corresponds to the total number of expert networks. A token is the basic data unit processed by the model, representing an input text fragment. The expert network is the sub-network module in the MoE model responsible for calculating specific features.

[0052] The system collects the first or second numerical value generated for each element in the sequence. Following the order of the original input tokens and the expert indices, the system organizes these values ​​into a two-dimensional matrix. The number of rows in this matrix equals the length of the input token sequence, and the number of columns equals the total number of expert networks. The value at each position in the matrix indicates whether the token in that specific row has selected the expert network corresponding to that specific column. The matrix position containing the first numerical value indicates that the token has selected that expert, and the matrix position containing the second numerical value indicates that the token has not selected that expert. The final generated two-dimensional matrix is ​​the expert mask table.

[0053] This application embodiment transforms a weighted generalized encoded sequence into a purely binary expert mask table. This table clearly records the selection relationship between each input token and each expert network, laying a precise data foundation for the subsequent generation of more advanced index auxiliary tables. This enables subsequent data rearrangement and batch matrix calculation to be performed efficiently based on explicit, non-zero expert selection relationships, thereby supporting the operation of the entire efficient MoE inference indexing mechanism.

[0054] In some embodiments, step 102 includes: Step 1021: According to the column direction of the expert mask table, the element values ​​of each column are accumulated sequentially. The accumulation calculation rule is to add the element value of the current row to the accumulated result of the previous row in the same column, and write the summation result to the corresponding position of the current row.

[0055] In this embodiment, the column direction refers to processing data along the vertical direction (i.e., from top to bottom) in a two-dimensional matrix data structure. Accumulation calculation is a mathematical operation that involves adding a series of values ​​one by one to obtain a cumulative sum. The calculation rule is a preset logic or formula followed when performing a specific operation. The current row refers to the matrix row being processed during traversal. The same column refers to the vertical data sequence consisting of all elements with the same horizontal coordinate (column index) as the current row element. The previous row refers to the row above and immediately adjacent to the current row. The accumulation result refers to the sum of all processed values ​​from the beginning of the sequence to the specified position. The summation result refers to the value obtained by adding two or more values. The corresponding position refers to the position in the output matrix that has the exact same row and column indices as the elements in the input data source matrix.

[0056] The NPU reads the expert mask table, which is a two-dimensional matrix. The system traverses the matrix column by column. For each column, starting from the second row, the system processes each row's elements sequentially. During processing, the system retrieves the element value of the current row in that column and reads the accumulated result of the previous row in the same column. The system adds these two values. The sum is then written to the first auxiliary index table at the exact same position as the row and column index of the currently processed element. This process is executed sequentially from top to bottom within each column.

[0057] Step 1022: For the first row element of each column, the summation result is the value of the first row element itself.

[0058] In this embodiment, the first row element refers to the first element of each column in the two-dimensional matrix, that is, the element with the smallest row index. The numerical value refers to the data value stored at a specific location in the matrix.

[0059] Before performing the column-wise accumulation in step 1021, or as a special case, the system needs to process the first row of each column. For each column of the expert mask table, the system locates its first row element. The system directly uses the value of that first row element itself as the accumulation result for that column in the first row. The system writes this value into the first auxiliary index table at the position corresponding to that first row element. This provides an initial base value for the accumulation calculation of subsequent rows.

[0060] Step 1023: Traverse all rows and columns of the expert mask table, complete the first type of prefix sum operation, and generate the first auxiliary index table.

[0061] In this embodiment, traversal refers to accessing each element in a data structure (such as a matrix) in a certain order (e.g., row-first, column-second, or column-first, row-second). All rows refer to all rows in the matrix from the first row to the last row. All columns refer to all columns in the matrix from the first column to the last column.

[0062] The system accesses each matrix element in the expert mask table in a predetermined order (e.g., traversing all columns first, then traversing all rows within each column). For each element, the system applies defined accumulation rules to calculate its sum based on whether its row is the first row of a column. The system then fills the corresponding position in the newly created first auxiliary index table with the accumulated result of each element. When the last element in the expert mask table has been processed and its accumulated result has been written to the first auxiliary index table, the system completes the entire first-type prefix sum operation and generates the complete first auxiliary index table.

[0063] Step 1024: The first auxiliary index table is a two-dimensional matrix with the same dimension as the expert mask table, where the value at each position represents the cumulative sum of the number of tokens selected by the same expert from the beginning of the sequence to the current position, and the last row of the first auxiliary index table constitutes the total number of tokens selected by each expert in the current sequence.

[0064] In this embodiment, dimension refers to a parameter describing the shape of an array or matrix, typically the number of rows and columns for a two-dimensional matrix. A two-dimensional matrix is ​​a rectangular array formed by arranging data in both row and column directions. In this context, sequence refers to the sequence of input tokens, corresponding to the row direction of the matrix. Cumulative sum refers to the continuous summation of all relevant values ​​from the start of the sequence to the current point. Last row refers to the last row of the two-dimensional matrix. Total number of tokens refers to the total number of times a particular expert was selected throughout the entire input sequence.

[0065] The system-generated first auxiliary index table is a two-dimensional matrix with the same number of rows and columns as the input expert mask table. In this table, the element in row i and column j represents the total number of times expert j was selected from the first token to the i-th token in the input token sequence (i.e., the first i rows). When the system reads the last row of the first auxiliary index table, the value in column j represents the total number of times expert j was selected by all tokens in the entire input sequence. The system extracts this last row to form a vector representing the total number of tokens selected by each expert in the current sequence.

[0066] This application embodiment generates a first auxiliary index table by transforming an expert mask table that records the sparse mapping relationship between each token and an expert. The core value of this table lies in its column-wise prefix sum characteristic, which allows the system to directly obtain the cumulative number of times each expert has been selected up to a given position by querying the value at any position in the table. In particular, its last row directly provides the total number of activations for each expert, which provides a crucial quantitative basis and offset calculation foundation for subsequent data rearrangement and expert batch calculation. This supports the transformation from an inefficient token-by-token calculation mode to an efficient expert-by-expert batch calculation mode, improving the computational parallelism and efficiency of the NPU in large-scale model MoE inference tasks.

[0067] In some embodiments, step 102 further includes: Step 1025: In the scenario of parallel inference of multiple batches of data, obtain the first auxiliary index table corresponding to each batch of data.

[0068] In this embodiment, batch data is a collection of multiple input data samples processed in parallel by the system during neural network inference. The first auxiliary index table is an expert mask accumulation table generated by prefix sum operation based on the routing results of all tokens in each batch. The data in each row represents the cumulative number of tokens selected by each expert from the starting token of the batch to the current token.

[0069] When processing multiple batches of data in parallel, the system first performs routing calculations and index table generation for each independent batch. Specifically, for each batch, the system generates a corresponding expert mask table based on the gated network outputs (i.e., generalized one-hot encoding) of all tokens within it. Subsequently, the system performs a first-type prefix sum operation on the token sequence dimension of this expert mask table, i.e., row-by-row accumulation, thereby generating an independent first auxiliary index table for each batch. This step ensures that the system can separately record and manage the cumulative state of the token-expert mapping relationship within each batch.

[0070] Step 1026: Extract the last row of data from each of the first auxiliary index tables to form a sub-table describing the number of tokens selected by experts in each batch.

[0071] In this embodiment of the application, the sub-table is a two-dimensional data table, whose rows correspond to the batch index and columns correspond to the expert index. Each element value in the table represents the total number of times a specific expert is selected by all tokens in the corresponding specific batch data.

[0072] After obtaining the first auxiliary index table corresponding to each batch of data, the system sequentially accesses the last rows of these tables. Since each row of the first auxiliary index table represents the cumulative number of selected tokens from the start of the batch to the current token, its last row represents the total number of tokens selected for each expert after all tokens have been selected within that batch. The system extracts these last rows and organizes them according to the order of their respective batches, thus forming a set consisting of multiple sub-tables. Each sub-table precisely describes the number of activated tokens for each expert in a single batch of data.

[0073] Step 1027: Stack the multiple sub-tables according to the batch processing order to form a three-dimensional data block. Perform the first type of prefix sum operation on the three-dimensional data block in the batch processing dimension to generate a cumulative table of the total number of tokens corresponding to each batch processing expert. The second auxiliary index table of the cumulative table of the total number of tokens corresponding to each batch processing expert describes the total number of tokens selected by each expert in the entire multi-batch processing sequence, and the intermediate layer data of the second auxiliary index table is used to indicate the cumulative number of tokens selected by all experts before a specific batch processing.

[0074] In this embodiment, the three-dimensional data block is a data structure whose three dimensions represent the batch index, expert index, and numerical value, respectively. The second auxiliary index table is a two-dimensional data table, whose rows correspond to the batch index and columns correspond to the expert index. Each element value in the table represents the cumulative total number of tokens selected by a specific expert in all processed batches from the first batch to the current batch.

[0075] The system stacks all generated sub-tables in the third dimension (batch processing dimension) according to the batch processing order, forming a three-dimensional data block. Next, the system performs a first-type prefix sum operation on this three-dimensional data block in the batch processing dimension. This operation accumulates layer by layer along the batch processing order: the data of the second layer (second batch) is added to the data of the first layer (first batch), the data of the third layer is added to the sum of the first two layers, and so on. After the operation is completed, a cumulative table of the total number of tokens corresponding to each batch processing expert is generated, i.e., the second auxiliary index table. The last row of this table describes the total number of tokens selected for each expert throughout the entire multi-batch processing sequence. Simultaneously, each row before the last row (intermediate layer data) in this table accurately records the cumulative number of tokens selected for each expert when reaching the corresponding batch processing index, providing global offset information for subsequent cross-batch data rearrangement and expert calculation.

[0076] This application embodiment aggregates and organizes sparse expert selection information scattered across various batches and tokens into a structured cumulative table through hierarchical prefix sum operations. These index tables enable the system to shift from a token-by-token processing perspective to an expert-centric batch processing perspective. Specifically, the system can use a second auxiliary index table to quickly determine the total number of tokens each expert needs to process and locate the starting memory address of its processed data. Simultaneously, combined with the first auxiliary index table, the system can further pinpoint the specific token data belonging to that expert within each batch. This achieves the merging and rearrangement of computation requests from multiple batches and tokens for a small number of experts, thereby transforming a large number of fragmented GEMV computations into more efficient and parallel GEMM computations, significantly improving the NPU's computational efficiency and hardware utilization in concurrent large-model MoE inference scenarios.

[0077] In some embodiments, step 103 includes: Step 1031: The second auxiliary index table is used as a one-dimensional vector, the length of which is equal to the total number of expert networks.

[0078] In this embodiment, a one-dimensional vector is a data structure whose elements are arranged in a linear order and can be accessed using only one index. The total number of expert networks refers to the number of all expert networks (e.g., small Transformer blocks) in the hybrid expert model.

[0079] The system reads the second auxiliary index table. This table is a list recording the total number of tokens selected for each expert network. The system converts this list data structure into a one-dimensional vector. The dimension (i.e., length) of this one-dimensional vector is set to be equal to the total number of expert networks in the model. After the conversion, each position (index) in the vector corresponds to a specific expert network, and the value stored at that position is the total number of tokens selected for that expert.

[0080] Step 1032: The element values ​​in the one-dimensional vector are accumulated sequentially.

[0081] In the embodiments of this application, cumulative calculation is a mathematical operation that refers to adding a series of values ​​one by one to obtain their sum or a sequence of partial sums.

[0082] The system initializes an accumulator variable to temporarily store intermediate accumulation results. Following the index order of the elements in the one-dimensional vector, the system accesses each element sequentially, starting from the first element. For the currently accessed element, the system plans to add its value to the sum of all previously accessed elements.

[0083] Step 1033: The rule for the cumulative calculation is to add the current element value to the cumulative result of the previous element, and write the summation result into the position of the current element.

[0084] In this embodiment, the system performs a specific accumulation operation. For the element currently being processed (not the first element), the system reads the accumulation result of the previous element from the accumulator. The system adds the value of the current element to the accumulation result of the previous element. After the operation is complete, the system writes the sum to the storage location corresponding to the current element in the vector, overwriting its original value. At the same time, the system updates the accumulator variable, using the current sum as the new accumulation result, in preparation for processing the next element.

[0085] Step 1034: For the first element, the summation result is the value of the first element itself.

[0086] In this embodiment, when the system begins processing the first element of a one-dimensional vector, since there are no previous elements, there is no accumulated result for the previous element. The system directly uses the value of this first element itself as its accumulated result. The system writes this value to the element's storage location in the vector. Simultaneously, the system initializes the accumulator variable to this value, serving as the starting point for subsequent element accumulation calculations.

[0087] Step 1035: Traverse all elements of the one-dimensional vector, complete the second type of prefix sum operation, and generate the third auxiliary index table. The third auxiliary index table is a one-dimensional vector with the same length as the second auxiliary index table, where the value at each position represents the cumulative sum of the total number of tokens selected by all experts from the first expert to the current expert. The third auxiliary index table is used to provide a global starting address offset for the computational data of each expert network in subsequent steps.

[0088] In this embodiment, the system executes the cumulative calculation process described in steps 1032 to 1034 in order from the first element to the last element. After all elements have been processed, the original one-dimensional vector (the second auxiliary index table) is transformed into a new vector, namely the third auxiliary index table. The length of this new vector is the same as the original vector. The value stored at the i-th position (i starts counting from 0) in the vector represents the cumulative sum of the total number of tokens selected by the i+1 experts from expert 0 to expert i (inclusive). After generating this table, the system stores it. The system will use this table in the subsequent expert network batch calculation stage. Specifically, for the i-th expert network, the global starting address offset of the input data required for its calculation can be directly determined by the value at the (i-1)-th position in this table (or 0, when i=0), thereby achieving efficient data location and loading.

[0089] This application provides a cumulative token count from the start of the model up to each expert, laying a crucial data addressing foundation for subsequent parallel computation of the expert network. Combined with other auxiliary index tables, the entire mechanism restructures the original token-by-token sparse computation mode into a dense batch computation mode per expert. This significantly improves the computational parallelism and efficiency of the neural network processor, especially when handling high-concurrency requests. Simultaneously, this mechanism reduces the need for pre-allocated memory space, and its implementation primarily relies on basic comparison and addition operations, requiring no dedicated hardware circuitry, thus exhibiting good versatility and practicality.

[0090] In some embodiments, step 104 includes: Step 1041: Obtain the original input feature data matrix, where the rows of the input feature data matrix correspond to the token sequence and the columns correspond to the feature dimensions.

[0091] In this embodiment, the input feature data matrix is ​​a two-dimensional data structure, where rows represent individual tokens in the input sequence and columns represent the feature vector dimensions of each token. This matrix represents the input data of a layer in a neural network model (such as Transformer), containing feature information of all tokens that need to be processed by the MoE layer.

[0092] The system first loads the raw input feature data matrix from memory or the output of the previous computational layer. The matrix's dimensions are (sequence length, hidden layer dimension). The system verifies the matrix's data format and memory layout to ensure it meets the input requirements of subsequent computational modules (such as routing networks). This step marks the beginning of the data rearrangement process, providing source data for subsequent index-based token selection and rearrangement.

[0093] Step 1042: Based on the identifier of the target expert, query the third auxiliary index table to obtain the global data offset of the expert level.

[0094] In this embodiment, the identifier of the target expert is a number or index value used to uniquely identify a specific expert network in the MoE layer. The third auxiliary index table is a data structure generated during the preprocessing stage. It stores the total number of tokens selected for each expert across all batches and all tokens, and is typically organized in the form of a prefix sum for quickly calculating the global offset of the expert level.

[0095] The system queries the third auxiliary index table based on the identifier of the target expert (e.g., expert ID=2). This table is a one-dimensional array with a length equal to the total number of experts. The query operation is performed directly using the expert identifier as the array index. The retrieved value represents the cumulative total number of tokens processed by all experts preceding the target expert (i.e., experts with IDs less than the current target expert). This value is the global data offset at the expert level, indicating the base address of the starting storage area for all token data belonging to this target expert in the final rearranged data buffer.

[0096] Step 1043: Based on the identifier of the current batch and the identifier of the target expert, query the intermediate layer data of the second auxiliary index table to obtain the local data offset of the batch processing level.

[0097] In this embodiment, the identifier of the current batch is a number used to distinguish different batches of input data. The intermediate layer data of the second auxiliary index table refers to the column data in the second auxiliary index table corresponding to a specific expert identifier. The second auxiliary index table itself is a two-dimensional table, with rows corresponding to batches and columns corresponding to experts. Each row is the prefix sum of the number of tokens selected for each expert in the current batch, used to calculate the offset of the batch level.

[0098] The system locates a specific cell in the second auxiliary index table based on the identifier of the currently being processed batch and the identifier of the target expert. Specifically, the system uses the batch identifier to locate the row and the expert identifier to locate the column. The retrieved value represents the cumulative number of tokens processed by all experts (in expert ID order) preceding the target expert in the current batch. This value is the local data offset at the batch level. It is an offset that further locates the starting position of the token data belonging to the target expert within the current batch, based on the global offset at the expert level.

[0099] Step 1044: For the target expert and the current batch, query the corresponding column of the first auxiliary index table to obtain the order index of the token level.

[0100] In this embodiment, the corresponding column of the first auxiliary index table refers to the column of data in the first auxiliary index table that corresponds to the target expert identifier. The first auxiliary index table is a two-dimensional table, where rows correspond to tokens and columns correspond to experts. Each element in the table is a binary mask or a compressed sequential index, indicating whether the current token has selected the expert, and if so, which token in the current batch has been selected for that expert.

[0101] For the current target expert and all tokens contained in the current batch, the system queries the column corresponding to the target expert in the first auxiliary index table. The query operation iterates through all rows (i.e., tokens) belonging to the current batch in that column. For each token, the query result is an index value. If the token is not selected by the target expert, the value may be 0 or a special flag; if it is selected, the value is a positive integer indicating which index of the token is selected within the current batch. This value is the order index of the token hierarchy.

[0102] Step 1045: Combining the global data offset, the local data offset, and the order index of the token level, calculate the specific storage location in the rearranged data buffer for each token data selected by the target expert in the original input feature data matrix.

[0103] In this embodiment, the rearranged data buffer is a contiguous storage area pre-allocated in memory, specifically used to store the rearranged feature data. Its organization is as follows: first, it is arranged in expert ID order; within each expert, it is arranged in batch order; and finally, within each batch, it is tightly arranged according to the order in which tokens were selected.

[0104] The system calculates the final storage address in the rearranged data buffer for each selected token in the current target expert and the current batch. The calculation method combines the three-level offsets obtained in steps 1042, 1043, and 1044. The specific formula is: Final storage location = Global data offset + Local data offset + Token hierarchy order index. The global data offset determines the starting point of the expert data block, the local data offset determines the starting point of the batch data within the expert block, and the token order index determines the precise location of the token within the batch block. The system uses this calculation to generate a unique target memory address for each selected token.

[0105] Step 1046: Copy the corresponding row data in the input feature data matrix to the corresponding position in the rearranged data buffer according to the specific storage location.

[0106] In this embodiment, the system performs a data transfer operation based on the calculated specific storage location. The system reads the entire row of feature data corresponding to the token selected by the target expert from the original input feature data matrix. Then, the system writes this feature data to the corresponding address calculated in step 1045 in the rearranged data buffer. This copying operation ensures that the feature data of the original token is accurately placed in a contiguous memory space organized around the expert.

[0107] Step 1047: Traverse all experts and all batches to complete the data rearrangement operation. The rearranged feature data is stored continuously according to the expert identifier, and the token data corresponding to the same expert is closely arranged.

[0108] In this embodiment, the system performs the complete rearrangement operation through nested loops. The outer loop iterates through all expert identifiers from 1 to N. For each expert, the inner loop iterates through all batch identifiers from 1 to B. In each round of the inner loop, the system executes steps 1042 to 1046 to locate and move the token data for the current expert and the current batch. When all experts and all batches have been processed, the data rearrangement operation is complete. At this point, the data in the rearranged data buffer is stored continuously according to the expert identifiers, and the token data corresponding to the same expert from different batches are also closely arranged together, forming an ideal data layout suitable for batch matrix multiplication (GEMM) calculations.

[0109] This application embodiment utilizes three pre-generated auxiliary index tables to dynamically and precisely rearrange the input data, originally organized in token sequence order, into contiguous data blocks organized around each expert. This data layout transformation lays a crucial foundation for subsequent expert computation to shift from an inefficient token-by-token vector multiplication mode to a highly parallel batch matrix multiplication mode for each expert.

[0110] In some embodiments, step 105 includes: Step 1051: Iterate through all expert identifiers from the first expert to the last expert.

[0111] In this embodiment, the expert identifier is a number or index used to uniquely distinguish different expert networks in the hybrid expert model. Traversal refers to the system or computer program accessing and processing each expert identifier sequentially according to a preset order (e.g., from 1 to N, where N is the total number of experts).

[0112] The system initiates a loop control flow, with the expert identifier as the iteration variable. The loop starts with the first preset expert identifier (e.g., identifier 1), processes one expert identifier in each iteration, and automatically updates the iteration variable to the next expert identifier after completing all processing for the current expert identifier. This loop continues to execute until the last expert identifier (e.g., identifier N) is processed, at which point the loop terminates.

[0113] Step 1052: For the current expert identifier, query the last row of the second auxiliary index table to obtain the total number of tokens that the current expert needs to process.

[0114] In this embodiment, the second auxiliary index table is a pre-calculated data structure whose last row stores a summary of the number of tokens each expert needs to process in the entire processing batch. A query refers to the system retrieving the corresponding value from this data structure based on a given index (the current expert's identifier).

[0115] When processing each expert identifier in a loop, the system uses the currently processed expert identifier as the index key. The system accesses the data storage area of ​​the second auxiliary index table, locates the last row of the table, and finds the corresponding data cell in that row based on the current expert identifier. The system reads the value stored in that data cell, which is the total number of tokens that the current expert network needs to participate in the computation of the entire input sequence.

[0116] Step 1053: Based on the total number of tokens and the feature dimension, extract all token feature data belonging to the current expert from the rearranged feature data buffer to form the input feature matrix of the current expert.

[0117] In this embodiment, the feature dimension refers to the length of the feature vector of each token. The rearranged feature data buffer is a memory area that stores the feature data of all tokens that have been reorganized according to the expert selection relationship. Extraction refers to the system copying a specific data block from this buffer. The input feature matrix is ​​a two-dimensional array with the number of rows equal to the total number of tokens processed by the current expert and the number of columns equal to the feature dimension. Each row represents the feature vector of a token belonging to that expert.

[0118] The system uses the total number of tokens obtained and the known feature dimensions to calculate the total amount of data to be read from the rearranged feature data buffer. Based on predetermined data layout rules (e.g., each expert's data is stored contiguously in the buffer, and the order matches the expert's identifier order), and combining the current expert identifier with the total number of tokens previously processed by the expert (obtainable by querying the first auxiliary index table), the system calculates the starting memory address in the buffer for all token feature data belonging to the current expert. Starting from this address, the system reads bytes of the calculated total data and organizes this data in a matrix format to form an input feature matrix specific to the current expert.

[0119] Step 1054: Obtain the weight matrix of the expert network corresponding to the current expert identifier.

[0120] In this embodiment, the weight matrix of the expert network is a model parameter matrix associated with a specific expert network, stored in the memory of the neural processing unit or a specific cache, and is used to transform the input features in matrix multiplication.

[0121] The system addresses the pre-loaded model parameter storage area based on the current expert identifier. By mapping the current expert identifier to a base address or lookup table, the system locates the storage location of the weight matrix corresponding to that expert network. The system then reads all the data from this storage location and prepares it for use by subsequent matrix multiplication units.

[0122] Step 1055: Perform a general matrix multiplication operation on the input feature matrix of the current expert and the weight matrix of the expert network to obtain the output feature matrix of the current expert. The general matrix multiplication operation is performed in batches on the matrix calculation unit of the neural processing unit. Repeat the above process until all expert networks have completed the calculation. The output feature matrices of all expert networks are stored in the output buffer according to the expert identifier order.

[0123] In this embodiment, general matrix multiplication is a basic linear algebra operation used to calculate the product of two matrices. The matrix computation unit of the neural processing unit is a circuit module in the hardware specifically optimized for efficiently performing large-scale matrix multiplications. Batch execution means that the computation unit processes the multiplication of the entire input feature matrix and weight matrix at once, rather than processing row by row or element by element. The output feature matrix is ​​the result of the matrix multiplication, and its dimension is determined by the number of rows in the input feature matrix and the number of columns in the weight matrix. The output buffer is a reserved memory area for temporarily storing the calculation results.

[0124] The system submits the constructed input feature matrix and the acquired weight matrix as operands to the matrix computation unit of the neural processing unit. The matrix computation unit receives these two matrices and initiates its internal parallel computing circuitry to perform a complete general matrix multiplication operation, generating the output feature matrix of the current expert. After calculation, the system writes the output feature matrix to a predetermined location in the output buffer, determined by the current expert's identifier, to ensure that all experts' outputs are arranged in identifier order. The system then returns to the loop start point of step 1051 to process the next expert identifier until the loop condition indicates that all experts have been processed.

[0125] This application's embodiments transform the highly serialized processing mode in hybrid expert models, which originally required selecting and calculating experts individually for each token, into a mode of batch matrix calculation per expert. This mechanism first establishes a batch calculation framework by traversing expert identifiers. Then, it uses a pre-generated index table to accurately and efficiently collect all token data belonging to each expert, forming a regular input matrix. Finally, it calls the dedicated matrix calculation unit of the neural processing unit to perform an efficient batch matrix multiplication on each expert. This significantly improves the utilization and parallelism of computing resources, transforming the core computation from a large number of small vector operations to a small number of large matrix operations, thereby greatly accelerating the inference process of large-scale hybrid expert layers, especially in high-concurrency scenarios.

[0126] In some embodiments, step 106 includes: Step 1061: Obtain the gating weight matrix output by the router. The rows of the gating weight matrix correspond to the token sequence, the columns correspond to the expert network, and the matrix has a non-zero weight value only at the selected expert position.

[0127] In this embodiment, the router refers to the gated network module in the MoE (Moment of Excel) model structure. Its function is to calculate and assign weights based on the characteristics of the input tokens to select a minority of expert networks to participate in the computation. The gated weight matrix is ​​a two-dimensional data structure, where the row dimension corresponds to each token in the input sequence and the column dimension corresponds to all available expert networks. This matrix is ​​sparse, with non-zero values ​​only at the positions corresponding to the minority experts selected for each token (determined by the Top-K operation). These non-zero values ​​are the routing weights.

[0128] The system obtains the gating weight matrix from the router's output interface. This matrix is ​​the direct result of the router processing all input tokens in the current batch. Each row of the matrix uniquely corresponds to an input token, and each column uniquely corresponds to an expert network. The element values ​​in the matrix represent the weights assigned by the router to the corresponding token and expert combination. According to the sparse activation property of MoE, for each token (i.e., each row of the matrix), only K elements (K is usually much smaller than the total number of experts) have non-zero weight values, and the remaining elements have zero values. The system stores this matrix in memory to provide the data foundation for the subsequent weighted fusion step.

[0129] Step 1062: Perform an inverse rearrangement operation on all expert output feature matrices stored in the output buffer after the batch general matrix multiplication operation is completed. The inverse rearrangement operation is the reverse process of the data rearrangement operation, which restores the data stored in the expert order to the original token order based on the first auxiliary index table, the second auxiliary index table and the third auxiliary index table.

[0130] In this embodiment, batch general matrix multiplication refers to the system performing batch matrix multiplication calculations on rearranged input data according to expert dimensions. The output buffer is a specific area in system memory used to temporarily store the results of batch general matrix multiplication operations. Reverse rearrangement is a data permutation operation, the reverse of data rearrangement. Data rearrangement reorganizes data originally organized in token order into an expert-ordered arrangement based on routing results to adapt to batch general matrix multiplication operations. The purpose of reverse rearrangement is to restore the result data calculated in expert order to its original token-ordered organizational form.

[0131] The system reads the expert output feature matrix stored in the output buffer. This matrix is ​​generated by the expert network performing batch general matrix multiplication on the rearranged input data. The system executes the inverse rearrangement algorithm, the core logic of which is the reverse process of the data rearrangement algorithm. Based on the mapping relationships recorded in the pre-generated and stored first, second, and third auxiliary index tables, the system accurately determines the original token position corresponding to each calculation result (i.e., the output vector after each token is processed by a certain expert) in the result matrix stored in the expert sequence. Then, the system extracts these calculation results from the expert sequence storage layout, rearranges them according to the original token sequence, writes them into a new buffer or overwrites the original input buffer, thereby generating an output feature matrix arranged in token index order.

[0132] Step 1063: Based on the position of each non-zero weight value in the gating weight matrix, locate the token and expert output feature vector corresponding to the inverse rearrangement operation.

[0133] In this embodiment, a non-zero weight value refers to an element in the gating weight matrix whose value is not zero. Each non-zero weight value is associated with a specific token and a specific selected expert. The expert output feature vector refers to the row vector (or column vector, depending on the data layout) in the output feature matrix corresponding to a specific token and a specific expert after the inverse rearrangement operation. It represents the processing result of the expert network for that token.

[0134] The system iterates through the acquired gating weight matrix. For each non-zero weight value in the matrix, the system parses its row and column indices. The row index identifies the token to which the weight belongs, and the column index identifies the expert to which the weight is assigned. Subsequently, the system uses the parsed token index to locate the data row of the corresponding token in the output feature matrix generated after the inverse rearrangement operation in step 1062, where the token order has been restored. Simultaneously, since each token may be processed by multiple experts, the system needs to further select the output feature vector of the expert corresponding to the current non-zero weight value from the multiple expert outputs corresponding to that token, based on the column index (expert identifier). This step establishes a precise correspondence between each routing weight and its weighted target data.

[0135] Step 1064: Perform a scalar multiplication operation between the non-zero weight value and the corresponding expert output feature vector.

[0136] In the embodiments of this application, scalar multiplication is a basic arithmetic operation, which refers to multiplying a scalar (a single numerical value) by each element of a vector to obtain a new vector.

[0137] For each pair of non-zero weight values ​​and expert output feature vectors, the system performs a scalar multiplication operation. Specifically, the system uses the non-zero weight value as a scalar multiplier and the corresponding expert output feature vector as the multiplicand. The arithmetic unit multiplies the weight scalar with the feature values ​​in each dimension of the vector sequentially, generating a new vector of the same dimensions. This new vector represents the expert contribution adjusted by the expert weight. All such weighting operations are performed independently in parallel or sequentially, scaling the contribution of each selected expert to each token independently.

[0138] Step 1065: Summate the multiple expert output feature vectors corresponding to the same token that have undergone scalar multiplication to obtain the final output feature vector of the token.

[0139] In this embodiment, for each unique token, the system collects all weighted expert output feature vectors calculated for that token in step 1064 (one vector for each expert selected by the token). Then, the system performs an element-wise summation operation on these vectors. That is, the first element of the first vector is added to the first element of the second vector, the first element of the third vector, and so on, to obtain the first element of the resulting vector, and so on, performing the same operation on all dimensions. The result of this summation operation is a single feature vector that integrates the weighted calculation results of all selected expert networks for that token, forming the final output representation of the token after processing by the current MoE layer.

[0140] Step 1066: Perform the weighted summation process on all tokens to combine them into the final output feature matrix of the large language model inference layer.

[0141] In this embodiment, the system iteratively executes the weighted summation process defined in steps 1063 to 1065 for each input token in the current batch. After processing each token, the final output feature vector of that token is arranged in order. When all tokens have been processed, the system combines these ordered final output feature vectors in memory to form a complete two-dimensional matrix. The rows of this matrix correspond to the token order of the original input sequence, and the columns correspond to the feature dimensions; it is the final output feature matrix of the current MoE inference layer. This matrix will serve as the input to subsequent neural network layers and continue to participate in the forward inference computation of the large language model.

[0142] This application's embodiments efficiently perform data inverse rearrangement using a pre-generated index table and perform precise weighted summation based on a sparse gating weight matrix. This transforms the fusion of multi-expert outputs, which originally required dynamic lookup and computation for each token, into a sequence of scalar multiplications and vector additions that can be processed in parallel or efficiently in a pipelined manner. Ultimately, the system can efficiently and accurately fuse the sparse contributions of multiple experts to the same token, generating the final output of the MoE layer. While ensuring model accuracy, it significantly improves the computational efficiency and hardware utilization of large-model MoE inference in high-concurrency scenarios and reduces storage space requirements.

[0143] In some embodiments, this invention provides an efficient NPU indexing mechanism. To implement this indexing mechanism, some auxiliary index tables need to be generated during the router's operational phase. (See reference...) Figure 3 The process for generating the index auxiliary table in a multi-batch scenario is as follows: Step S1: Generate an expert mask table; Step S2: Generate the expert mask acc table to obtain the total number of tokens corresponding to each batch of experts; Step S3: Generate the total number of tokens for each batch of experts (acc table) to obtain the total number of tokens for each expert. Step S4: Generate the total number of tokens corresponding to each expert (acc table); Step S5: Generate an expert order table corresponding to the token.

[0144] The process of generating the expert mask table is as follows: Figure 4 As shown, the generalized one-hot encoding after softmax is compared with 0 to obtain a 01 mask. The expert mask at this point indicates whether a token has selected a particular expert.

[0145] The method for generating the expert mask (acc) table is prefix sum; see [example example]. Figure 5 The specific operation is as follows: add the first row to the second row, add the second row to the third row, and so on.

[0146] Generate an expert mask (acc) table for each batch. Extracting the last row of each batch yields the total number of tokens corresponding to each expert in that batch. The method for generating the acc table is to perform a prefix sum on the acc table for each batch's expert-to-total ... Figure 6 The last row of the resulting table is the total number of tokens for each expert.

[0147] The method for generating the total number of tokens (acc) for each expert is to perform a prefix sum on the total number of tokens for each expert. See the example below. Figure 7 .

[0148] If the hardware supports a table lookup operation, zeros can be removed by discarding useless values ​​from the expert mask acc table, generating a token-to-expert order table. See the example below. Figure 8 .

[0149] In multi-batch scenarios, the MoE processing flow can be divided into the following steps: route calculation and index auxiliary table generation, data rearrangement, expert batch GEMM calculation, and weighted fusion of output results. See [link to detailed flow] for more information. Figure 9 .

[0150] Weighted fusion, compared to the data rearrangement stage, only adds a topk-weight table. For each token, the topk experts selected by the index are then summed with the topk-weight table using weighted methods. The index experts are selected in the same way as the data rearrangement stage; see [link to process details] for details. Figure 10 The number offset of expert levels is determined by the total number of tokens corresponding to each expert (acc table). The number offset of batch levels under each expert is determined by the total number of tokens corresponding to each batch expert (acc table). Finally, the specific position of the final token level is obtained by the token-to-expert order table.

[0151] For expert batch GEMM calculations, iterate through experts 1-n and perform GEMM calculations sequentially. The number of matrix multiplications for each expert is determined using the last row of the `acc` table (the total number of tokens for each expert). The number offset for each expert level is determined using the `acc` table, which facilitates finding the starting address of each expert's matrix multiplication.

[0152] Based on the foregoing embodiments, this application provides an indexing device for a large language model. The device includes various units and modules included in each unit, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0153] Figure 11 This application provides a schematic diagram of the composition structure of an indexing device for a large language model, as shown in the embodiments of the present application. Figure 11 As shown, the indexing device 20 of the large language model includes: Processing module 201 is used to generate an expert mask table based on the comparison result between the generalized coding sequence output by the router and the preset value; Perform a first type of prefix sum operation on the expert mask table to generate a first auxiliary index table, and integrate the last row data of the first auxiliary index table in multiple batch processing scenarios to generate a second auxiliary index table; Perform a second type of prefix sum operation on the last row of the second auxiliary index table to generate a third auxiliary index table; Based on the first auxiliary index table, the second auxiliary index table and the third auxiliary index table, the input feature data is rearranged to obtain rearranged feature data; Perform expert-level batch general matrix multiplication on the rearranged feature data; The output module 202 is used to perform a weighted fusion operation on the result of the batch general matrix multiplication operation based on the gating weights output by the router, so as to obtain the final output features of the large language model MOE architecture inference stage.

[0154] In some embodiments, the processing module 201 is further configured to: Each element value in the generalized encoded sequence is compared with zero; When the element value is greater than zero, the comparison result is marked as the first value; when the element value is equal to or less than zero, the comparison result is marked as the second value. The first value and the second value are arranged according to the original dimensions of the generalized coding sequence to form a two-dimensional matrix containing the first value and the second value. The two-dimensional matrix is ​​the expert mask table. The row dimension of the expert mask table corresponds to the length of the input token sequence, and the column dimension corresponds to the total number of expert networks. The expert mask table is used to indicate the selection relationship between each input token and each expert network. The first value represents that the corresponding expert is selected by the current token, and the second value represents that the corresponding expert is not selected by the current token.

[0155] In some embodiments, the processing module 201 is further configured to: According to the column direction of the expert mask table, the element values ​​of each column are accumulated sequentially. The accumulation calculation rule is to add the element value of the current row to the accumulated result of the previous row in the same column, and write the summation result into the corresponding position of the current row. For the first element of each column, the summation result is the value of the first element itself. Traverse all rows and columns of the expert mask table, perform the first type of prefix sum operation, and generate the first auxiliary index table; The first auxiliary index table is a two-dimensional matrix with the same dimensions as the expert mask table, where the value at each position represents the cumulative sum of the number of tokens selected by the same expert from the beginning of the sequence to the current position, and the last row of the first auxiliary index table constitutes the total number of tokens selected by each expert in the current sequence.

[0156] In some embodiments, the processing module 201 is further configured to: In a scenario where multiple batches of data are processed in parallel, the first auxiliary index table corresponding to each batch of data is obtained. Extract the last row of data from each of the first auxiliary index tables to form a sub-table describing the number of tokens selected by experts in each batch; Multiple sub-tables are stacked in batch processing order to form a three-dimensional data block; the first type of prefix sum operation is performed on the three-dimensional data block in the batch processing dimension to generate a cumulative table of the total number of tokens corresponding to each batch processing expert, and a second auxiliary index table for the cumulative table of the total number of tokens corresponding to each batch processing expert. The second auxiliary index table describes the total number of tokens selected by each expert in the entire multi-batch processing sequence, and the intermediate layer data of the second auxiliary index table is used to indicate the cumulative number of tokens selected by all experts before a specific batch processing.

[0157] In some embodiments, the processing module 201 is further configured to: The second auxiliary index table is used as a one-dimensional vector, and the length of the one-dimensional vector is equal to the total number of expert networks. The element values ​​in the one-dimensional vector are sequentially summed. The rule for the cumulative calculation is to add the current element value to the cumulative result of the previous element, and then write the summation result into the position of the current element. For the first element, the summation result is the value of the first element itself; Traverse all elements of the one-dimensional vector, perform the second type of prefix sum operation, and generate the third auxiliary index table. The third auxiliary index table is a one-dimensional vector with the same length as the second auxiliary index table, where the value at each position represents the cumulative sum of the total number of tokens selected by all experts from the first expert to the current expert. The third auxiliary index table is used to provide a global starting address offset for the computational data of each expert network in subsequent steps.

[0158] In some embodiments, the processing module 201 is further configured to: Obtain the original input feature data matrix, where the rows of the input feature data matrix correspond to the token sequence and the columns correspond to the feature dimensions; Based on the identifier of the target expert, query the third auxiliary index table to obtain the global data offset of the expert level; Based on the identifier of the current batch and the identifier of the target expert, query the intermediate layer data of the second auxiliary index table to obtain the local data offset of the batch level; For the target expert and the current batch, query the corresponding column of the first auxiliary index table to obtain the order index of the token level; By combining the global data offset, the local data offset, and the order index of the token level, the specific storage location of each token data selected by the target expert in the original input feature data matrix in the rearranged data buffer is calculated. According to the specific storage location, copy the corresponding row data in the input feature data matrix to the corresponding position in the rearranged data buffer; The data is rearranged by iterating through all experts and all batches. The rearranged feature data is stored continuously according to the expert identifier, and the token data corresponding to the same expert is closely arranged.

[0159] In some embodiments, the processing module 201 is further configured to: Iterate through all expert identifiers from the first expert to the last expert; For the current expert identifier, query the last row of the second auxiliary index table to obtain the total number of tokens that the current expert needs to process; Based on the total number of tokens and the feature dimension, extract all token feature data belonging to the current expert from the rearranged feature data buffer to form the input feature matrix of the current expert; Obtain the weight matrix of the expert network corresponding to the current expert identifier; The input feature matrix of the current expert is multiplied by the weight matrix of the expert network to obtain the output feature matrix of the current expert. The general matrix multiplication operation is performed in batches on the matrix calculation unit of the neural processing unit. The above process is repeated until all expert networks have been calculated. The output feature matrices of all expert networks are stored in the output buffer in the order of expert identifiers.

[0160] In some embodiments, the output module 202 is further configured to: Obtain the gated weight matrix output by the router. The rows of the gated weight matrix correspond to the token sequence, the columns correspond to the expert network, and the non-zero weight value is only found at the selected expert position. After the batch general matrix multiplication operation is completed, all expert output feature matrices stored in the output buffer are subjected to an inverse rearrangement operation. The inverse rearrangement operation is the reverse process of the data rearrangement operation, which restores the data stored in the expert order to the original token order based on the first auxiliary index table, the second auxiliary index table and the third auxiliary index table. Based on the position of each non-zero weight value in the gating weight matrix, locate the token and expert output feature vector corresponding to the inverse rearrangement operation; Perform a scalar multiplication operation between the non-zero weight value and the corresponding expert output feature vector; The final output feature vector of the token is obtained by summing the multiple expert output feature vectors corresponding to the same token that have undergone scalar multiplication. Perform the weighted summation process on all tokens to combine them into the final output feature matrix of the large language model inference layer.

[0161] This application's embodiments systematically transform the traditional, low-parallelism per-token general matrix-vector multiplication inference mode into a higher-parallelism per-expert batch general matrix multiplication mode by generating and utilizing a series of index auxiliary tables. This significantly improves the computational efficiency of the NPU at the MoE layer, especially in high-concurrency inference scenarios. Simultaneously, this mechanism optimizes the data layout in memory, eliminating the need to reserve maximum possible space for each expert when pre-allocating storage space; instead, it only needs to allocate space for the actually selected experts, thereby drastically reducing memory usage. The entire solution primarily relies on basic comparison and addition operations, has no special requirements for hardware circuitry, is highly versatile, and is easy to implement and deploy on existing NPU architectures.

[0162] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0163] It should be noted that, in the embodiments of this application, if the above-mentioned indexing method for the large language model is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0164] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0165] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0166] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0167] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0168] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0169] It should be noted that, Figure 12 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 12 As shown, the hardware entity of the computer device 700 includes: one or more processors 701, a communication interface 702, and a memory 703, wherein: Processor 701 typically controls the overall operation of computer device 700.

[0170] Communication interface 702 enables computer devices to communicate with other terminals or servers over a network.

[0171] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the computer device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704. Only one processor is shown in the figure; each processor 700 includes one or more cores.

[0172] It should be noted that the computer device may include multiple processors 701, and each processor 701 can interact with each other through aggregated communication methods such as all-to-all, all-gather, or all-reduce. The processors 701 may be central processing units (CPUs), graphics processing units (GPUs), embedded neural network processing units (NPUs), tensor processing units (TPUs), data processing units (DPUs), accelerated processing units (APUs), floating-point processing units (FPUs), or application-specific integrated circuits (ASICs). The processors may also be single-core or multi-core processors. The processor may consist of a CPU and hardware chips. The hardware chips may be ASICs, PLDs, or combinations thereof. The PLDs may be complex programmable logic devices (CPLDs), FPGAs, generic array logic (GALs), or any combination thereof. The processor can also be implemented using logic devices with built-in processing logic, such as FPGAs or digital signal processors (DSPs).

[0173] The communication interface 702 can be a wired interface or a wireless interface, used to communicate with other modules or devices. The wired interface can be an Ethernet interface, a local interconnect network (LIN), etc., and the wireless interface can be a cellular network interface or a wireless LAN interface, etc.

[0174] Memory 703 can be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 703 can also be volatile memory, which can be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synclink dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM), direct rambus DRAM (DRDRAM), and rambus DRAM.

[0175] The 704 bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc.

[0176] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0177] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0179] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0180] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0181] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0182] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0183] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An indexing method for a large language model, characterized in that, The method includes: An expert mask table is generated based on the comparison results between the generalized coded sequence output by the router and the preset values. Perform a first type of prefix sum operation on the expert mask table to generate a first auxiliary index table, and integrate the last row data of the first auxiliary index table in multiple batch processing scenarios to generate a second auxiliary index table; Perform a second type of prefix sum operation on the last row of the second auxiliary index table to generate a third auxiliary index table; Based on the first auxiliary index table, the second auxiliary index table and the third auxiliary index table, the input feature data is rearranged to obtain rearranged feature data; Perform expert-level batch general matrix multiplication on the rearranged feature data; Based on the gating weights output by the router, a weighted fusion operation is performed on the results of the batch general matrix multiplication operation to obtain the final output features of the MOE architecture inference stage.

2. The method according to claim 1, characterized in that, The process of generating an expert mask table based on the comparison result between the generalized coded sequence output by the router and a preset value includes: Each element value in the generalized encoded sequence is compared with zero; When the element value is greater than zero, the comparison result is marked as the first value; when the element value is equal to or less than zero, the comparison result is marked as the second value. The first value and the second value are arranged according to the original dimensions of the generalized coding sequence to form a two-dimensional matrix containing the first value and the second value. The two-dimensional matrix is ​​the expert mask table. The row dimension of the expert mask table corresponds to the length of the input token sequence, and the column dimension corresponds to the total number of expert networks. The expert mask table is used to indicate the selection relationship between each input token and each expert network. The first value represents that the corresponding expert is selected by the current token, and the second value represents that the corresponding expert is not selected by the current token.

3. The method according to claim 1, characterized in that, The step of performing a first type of prefix sum operation on the expert mask table to generate a first auxiliary index table includes: According to the column direction of the expert mask table, the element values ​​of each column are accumulated sequentially. The accumulation calculation rule is to add the element value of the current row to the accumulated result of the previous row in the same column, and write the summation result into the corresponding position of the current row. For the first element of each column, the summation result is the value of the first element itself. Traverse all rows and columns of the expert mask table, perform the first type of prefix sum operation, and generate the first auxiliary index table; The first auxiliary index table is a two-dimensional matrix with the same dimensions as the expert mask table, where the value at each position represents the cumulative sum of the number of tokens selected by the same expert from the beginning of the sequence to the current position, and the last row of the first auxiliary index table constitutes the total number of tokens selected by each expert in the current sequence.

4. The method according to claim 1, characterized in that, The step of integrating the last row data of the first auxiliary index table in multiple batch processing scenarios to generate a second auxiliary index table includes: In a scenario where multiple batches of data are processed in parallel, the first auxiliary index table corresponding to each batch of data is obtained. Extract the last row of data from each of the first auxiliary index tables to form a sub-table describing the number of tokens selected by experts in each batch; Multiple sub-tables are stacked in batch processing order to form a three-dimensional data block; the first type of prefix sum operation is performed on the three-dimensional data block in the batch processing dimension to generate a cumulative table of the total number of tokens corresponding to each batch processing expert, and a second auxiliary index table for the cumulative table of the total number of tokens corresponding to each batch processing expert. The second auxiliary index table describes the total number of tokens selected by each expert in the entire multi-batch processing sequence, and the intermediate layer data of the second auxiliary index table is used to indicate the cumulative number of tokens selected by all experts before a specific batch processing.

5. The method according to claim 1, characterized in that, The step of performing a second type of prefix sum operation on the last row of the second auxiliary index table to generate a third auxiliary index table includes: The second auxiliary index table is used as a one-dimensional vector, and the length of the one-dimensional vector is equal to the total number of expert networks. The element values ​​in the one-dimensional vector are sequentially summed. The rule for the cumulative calculation is to add the current element value to the cumulative result of the previous element, and then write the summation result into the position of the current element. For the first element, the summation result is the value of the first element itself; Traverse all elements of the one-dimensional vector, perform the second type of prefix sum operation, and generate the third auxiliary index table. The third auxiliary index table is a one-dimensional vector with the same length as the second auxiliary index table, where the value at each position represents the cumulative sum of the total number of tokens selected by all experts from the first expert to the current expert. The third auxiliary index table is used to provide a global starting address offset for the computational data of each expert network in subsequent steps.

6. The method according to claim 1, characterized in that, The step of rearranging the input feature data according to the first auxiliary index table, the second auxiliary index table, and the third auxiliary index table to obtain rearranged feature data includes: Obtain the original input feature data matrix, where the rows of the input feature data matrix correspond to the token sequence and the columns correspond to the feature dimensions; Based on the identifier of the target expert, query the third auxiliary index table to obtain the global data offset of the expert level; Based on the identifier of the current batch and the identifier of the target expert, query the intermediate layer data of the second auxiliary index table to obtain the local data offset of the batch level; For the target expert and the current batch, query the corresponding column of the first auxiliary index table to obtain the order index of the token level; By combining the global data offset, the local data offset, and the order index of the token level, the specific storage location of each token data selected by the target expert in the original input feature data matrix in the rearranged data buffer is calculated. According to the specific storage location, copy the corresponding row data in the input feature data matrix to the corresponding position in the rearranged data buffer; The data is rearranged by iterating through all experts and all batches. The rearranged feature data is stored continuously according to the expert identifier, and the token data corresponding to the same expert is closely arranged.

7. The method according to claim 1, characterized in that, The step of performing expert-level batch general matrix multiplication on the rearranged feature data includes: Iterate through all expert identifiers from the first expert to the last expert; For the current expert identifier, query the last row of the second auxiliary index table to obtain the total number of tokens that the current expert needs to process; Based on the total number of tokens and the feature dimension, extract all token feature data belonging to the current expert from the rearranged feature data buffer to form the input feature matrix of the current expert; Obtain the weight matrix of the expert network corresponding to the current expert identifier; The input feature matrix of the current expert is multiplied by the weight matrix of the expert network to obtain the output feature matrix of the current expert. The general matrix multiplication operation is performed in batches on the matrix calculation unit of the neural processing unit. The above process is repeated until all expert networks have been calculated. The output feature matrices of all expert networks are stored in the output buffer in the order of expert identifiers.

8. The method according to claim 1, characterized in that, The weighted fusion operation is performed on the results of the batch general matrix multiplication operation based on the gating weights output by the router to obtain the final output features of the large language model MOE architecture inference stage, including: Obtain the gated weight matrix output by the router. The rows of the gated weight matrix correspond to the token sequence, the columns correspond to the expert network, and the non-zero weight value is only found at the selected expert position. After the batch general matrix multiplication operation is completed, all expert output feature matrices stored in the output buffer are subjected to an inverse rearrangement operation. The inverse rearrangement operation is the reverse process of the data rearrangement operation, which restores the data stored in the expert order to the original token order based on the first auxiliary index table, the second auxiliary index table and the third auxiliary index table. Based on the position of each non-zero weight value in the gating weight matrix, locate the token and expert output feature vector corresponding to the inverse rearrangement operation; Perform a scalar multiplication operation between the non-zero weight value and the corresponding expert output feature vector; The final output feature vector of the token is obtained by summing the multiple expert output feature vectors corresponding to the same token that have undergone scalar multiplication. Perform the weighted summation process on all tokens to combine them into the final output feature matrix of the large language model inference layer.

9. An indexing device for a large language model, characterized in that, The device includes: The processing module is used to generate an expert mask table based on the comparison results between the generalized coded sequence output by the router and the preset values. Perform a first type of prefix sum operation on the expert mask table to generate a first auxiliary index table, and integrate the last row data of the first auxiliary index table in multiple batch processing scenarios to generate a second auxiliary index table; Perform a second type of prefix sum operation on the last row of the second auxiliary index table to generate a third auxiliary index table; Based on the first auxiliary index table, the second auxiliary index table and the third auxiliary index table, the input feature data is rearranged to obtain rearranged feature data; Perform expert-level batch general matrix multiplication on the rearranged feature data; The output module is used to perform a weighted fusion operation on the results of the batch general matrix multiplication operation based on the gating weights output by the router, so as to obtain the final output features of the inference stage of the large language model MOE architecture.

10. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the indexing method of the large language model according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps in the indexing method of the large language model as described in any one of claims 1 to 8.

Images