A method, processing device, electronic device, and storage medium for accelerating inference of MoE large language models on resource-constrained devices.

Abstract

This invention discloses a method, processing device, electronic device, and storage medium for accelerating inference using the MoE large language model on resource-constrained devices. The method includes: inferring from prompt word data using the MoE large language model; calculating expert scores during token generation to obtain expert activation patterns; constructing a dataset using binary data pairs of prompt word data and their corresponding expert activation patterns; training an expert activation prediction model using this dataset; performing initial inference on each inference request using the MoE large language model; predicting the corresponding expert activation pattern for each inference request at each layer of the MoE large language model using the trained expert activation prediction model; and selecting the corresponding expert from each layer of the MoE large language model for each inference request based on the predicted activation probability values, and preloading them into each layer all at once. This invention achieves efficient and high-quality expert selection.

Description

Technical Field

[0001] This invention relates to the field of large language model optimization inference technology, and in particular to a method, processing device, electronic device and storage medium for accelerating inference of MoE large language models on resource-constrained devices. Background Technology

[0002] Accelerating inference for MoE (Mixture of Experts) large language models on resource-constrained devices is a performance optimization problem that requires a comprehensive consideration of the balance between inference quality and inference efficiency.

[0003] There are two main solutions to the existing technology. One is to leverage the activation similarity exhibited by experts in adjacent MoE layers during single token generation to accelerate MoE inference through layer-level expert prediction and prefetching. The other is to achieve expert prediction and prefetching through expert activation matrices or token similarity matching.

[0004] However, both of the above schemes perform expert prediction and prefetching at the token level, without fully exploring expert activation similarity, resulting in high resource overhead. Furthermore, the expert activation errors obtained through similarity matching are relatively large, making high-quality inference difficult and challenging to guarantee a balanced inference performance. Summary of the Invention

[0005] The purpose of this invention is to provide a method, processing device, electronic device, and storage medium for accelerating inference of the MoE large language model on resource-constrained devices, in order to solve the problems of large expert activation errors and difficulty in ensuring a performance balance between high-quality and high-efficiency inference in existing inference acceleration technologies. The various technical effects of the preferred solutions among the many technical solutions provided by this invention are detailed below.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] This invention provides a method for accelerating inference of a MoE large language model on resource-constrained devices, comprising the following steps:

[0008] By reasoning about the prompt word data using the MoE large language model and calculating the expert score in the token generation process, the expert activation mode corresponding to the prompt word data is obtained.

[0009] The prompt word data and its corresponding expert activation pattern are used as binary data pairs to form a dataset, and the expert activation prediction model is trained using the dataset.

[0010] The MoE large language model is used to perform initial inference for each inference request, and the trained expert activation prediction model is used to predict the expert activation mode corresponding to each inference request at each layer of the MoE large language model.

[0011] Based on the predicted activation probability value, select the corresponding expert from each layer of the MoE large language model corresponding to each inference request from the predicted expert activation mode, and preload them into each layer at once to perform inference for each corresponding inference request.

[0012] In one or more embodiments, the k-th expert score of the l-th layer of the MoE large language model corresponding to the i-th prompt word data is... It can be expressed by the following formula:

[0013] ;

[0014] in, This indicates that it corresponds to an expert. Routing weights and These represent the number of experts at each layer of the MoE large language model and the number of layers in the MoE large language model, respectively. Indicates an indicator function, if Used for inference generation If the value is 1, then the value is 1; otherwise, the value is 0. The length of the response output obtained by reasoning from the i-th prompt word data.

[0015] In one or more embodiments, for the i-th prompt word data, the expert scores of the experts tracked by the indicator function at different layers of the MoE large language model collectively constitute the expert activation mode corresponding to the i-th prompt word data.

[0016] In one or more embodiments, the expert activation prediction model is independent of the MoE large language model and is used to predict the experts that need to be activated in all layers of the MoE large language model at once according to the corresponding inference request; the input of the expert activation prediction model is a prompt word, and the output is the expert activation pattern corresponding to the prompt word, which represents the importance probability of all experts being activated in all layers.

[0017] In one or more embodiments, the expert activation prediction model includes a pre-trained language model encoding module and a multilayer perceptron prediction module, wherein the pre-trained language model encoding module is used to capture semantic features in input prompt words and generate prompt word embeddings; the multilayer perceptron prediction module is used to learn the mapping relationship between prompt word embeddings and expert activation patterns.

[0018] In one or more embodiments, before performing initial inference for each inference request using the MoE large language model, the method further includes:

[0019] Non-expert parameters are stored in GPU memory, while expert parameters are loaded into CPU memory. Simultaneously, the trained expert activation prediction model is loaded into CPU memory for the MoE large language model.

[0020] In one or more embodiments, when training the expert activation prediction model using the dataset, the predicted expert activation pattern corresponding to each prompt word data is fitted to its true expert activation pattern by minimizing a hybrid loss function composed of cross-entropy loss and Huber loss weighted together, thereby obtaining the trained expert activation prediction model.

[0021] According to another aspect of the present invention, a processing apparatus is also provided, including a configuration module and a parsing module.

[0022] The configuration module is used to configure and edit the protocol file corresponding to the inference acceleration method of the MoE large language model on resource-constrained devices described above. The protocol file is binary format data. The parsing module is used to parse the protocol file into JSON format data.

[0023] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored thereon, and when executed, the computer program implements the above-described method for accelerating inference of a MoE large language model on a resource-constrained device.

[0024] According to another aspect of the present invention, an electronic device is also provided, comprising:

[0025] One or more processors;

[0026] A memory for storing one or more computer programs, and one or more processors for executing the one or more computer programs stored in the memory to enable the one or more processors to perform the inference acceleration method for the MoE large language model on resource-constrained devices as described above.

[0027] Implementing one of the above-described technical solutions of the present invention has the following advantages or beneficial effects:

[0028] This invention leverages the expert activation similarity of adjacent tokens generated in the early decoding stage within the same layer of the MoE large language model for expert prediction and prefetching, thereby accelerating inference of the MoE large language model on resource-constrained devices. To address the challenge of balancing inference efficiency and quality in expert prediction and prefetching, an expert activation pattern with cue word granularity is obtained through expert scores, achieving efficient and high-quality expert selection. Through a single expert prediction and prefetching process, superior inference performance of the MoE large language model is obtained. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0030] Figure 1 This is a flowchart of an inference acceleration method for a MoE large language model on a resource-constrained device according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of an electronic device structure according to an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer, various exemplary embodiments described below will be referenced to the accompanying drawings, which form part of the exemplary embodiments, illustrating various exemplary embodiments that may be used to implement the present invention. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. It should be understood that they are merely examples of processes, methods, and apparatuses consistent with some aspects of the present invention disclosed as detailed in the appended claims, and other embodiments may be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and spirit of the present invention.

[0033] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," etc., indicate the orientation or positional relationship based on the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the referred element must have a specific orientation, or be constructed and operated in a specific orientation. The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. The term "multiple" means two or more. The terms "connected" and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, integral connections, mechanical connections, electrical connections, communication connections, direct connections, indirect connections through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more of the related listed items. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0034] To illustrate the technical solution described in this invention, specific embodiments are described below, showing only the parts related to the embodiments of this invention.

[0035] Example 1:

[0036] like Figure 1 As shown, this invention provides a method for accelerating inference of a MoE large language model on resource-constrained devices, comprising the following steps:

[0037] S100. Reasoning is performed on the prompt word data using the MoE large language model. The expert score during the token generation process is statistically analyzed to obtain the expert activation mode corresponding to the prompt word data.

[0038] In this embodiment, the prompt word data is used to train the expert activation prediction model, and there are naturally multiple of them. When using the MoE large language model to perform inference on the input i-th prompt word data, several tokens are usually generated.

[0039] The MoE large language model is used to process the input data of the i-th prompt word. Perform reasoning to obtain the corresponding response output. .in, and These are the input and output lengths, respectively. During this process, an expert tracker is used to calculate the expert score corresponding to the token generation process, thus obtaining the prompt word data. Corresponding expert activation mode The expert score takes into account both the frequency and importance of expert selection.

[0040] In one or more embodiments, the k-th expert score of the l-th layer of the MoE large language model corresponding to the i-th prompt word data is... It can be expressed by the following formula:

[0041] ;

[0042] in, This indicates that it corresponds to an expert. Routing weights and These represent the number of experts at each layer of the MoE large language model and the number of layers in the MoE large language model, respectively. Indicates an indicator function, if Used for inference generation If the value is 1, then the value is 1; otherwise, the value is 0. The length of the response output obtained by reasoning from the i-th prompt word data.

[0043] Furthermore, for the i-th cue word data, the expert scores tracked by the experts at different layers of the MoE large language model through the indicator function collectively constitute the expert activation mode corresponding to the i-th cue word data. .

[0044] S200. The prompt word data and its corresponding expert activation pattern are used as binary data pairs to form a dataset, and the expert activation prediction model is trained using the dataset.

[0045] Based on the above embodiments, for input prompt words The expert scores of the experts at these different levels that were tracked. Together, they constitute the corresponding expert activation mode. , represented as .in, For the number of MoE layers, The number of experts at each level. Then, these cue word data... Its corresponding expert activation mode This will be further used as binary data pairs to form the dataset. , This represents the number of data items.

[0046] In one or more embodiments, the expert activation prediction model includes a pre-trained language model encoding module and a multilayer perceptron prediction module. The pre-trained language model encoding module is used to capture semantic features in the input prompt words and generate prompt word embeddings; the multilayer perceptron prediction module is used to learn the mapping relationship between the prompt word embeddings and the expert activation patterns.

[0047] It should be noted that the expert activation prediction model is independent of the MoE large language model and is used to predict the experts required to activate in all layers of the MoE large language model at once based on the corresponding inference request. Its input... Output the prompt word. This represents the corresponding expert activation pattern, indicating the importance probability of all experts being activated across all layers. It is a size of The matrix, For the number of MoE layers, The number of experts per floor.

[0048] Furthermore, using the aforementioned dataset The expert activation prediction model is trained end-to-end. By minimizing a hybrid loss function weighted by cross-entropy loss and Huber loss, the model is optimized for each prompt word. The corresponding prediction expert activation mode Its real expert activation mode By fitting the model, a well-trained expert activation prediction model is obtained.

[0049] Understandably, using a weighted combination of cross-entropy loss and Huber loss as a hybrid loss function can provide a stable and continuous measure of prediction error, enabling model parameters to converge toward the optimal direction. This improves the network's prediction accuracy, convergence stability, and generalization ability, while reducing training oscillations and increasing training efficiency.

[0050] In one or more embodiments, after the expert activation prediction model has been trained and before initial inference is performed on each inference request using the MoE large language model, the method further includes:

[0051] Non-expert parameters are stored in GPU memory, while expert parameters are loaded into CPU memory. At the same time, the trained expert activation prediction model is loaded into CPU memory for the MoE large language model.

[0052] The aforementioned non-expert parameters refer to parameters in the model that are not related to the calculations of the experts themselves and are shared across all inputs or all routing paths, including but not limited to embedding layer parameters, attention layer parameters, normalization layer parameters, output layer parameters, and router parameters in the MoE layer.

[0053] Understandably, offloading the expert parameters, which account for a large proportion of the MoE large language model, to CPU memory and retaining only non-expert parameters on the GPU can significantly reduce GPU memory requirements; placing the expert activation prediction model on the CPU further enables the efficient use of limited GPU resources.

[0054] S300. Use the MoE large language model to perform initial reasoning for each reasoning request, and use the trained expert activation prediction model to predict the expert activation mode corresponding to each reasoning request at each layer of the MoE large language model.

[0055] In this step, for each new inference request While using the MoE large language model for initial inference, the trained expert activation prediction model is used to predict the corresponding expert activation patterns. .

[0056] S400. Based on the predicted activation probability value, select the corresponding expert from each layer of the MoE large language model corresponding to each inference request from the predicted expert activation mode, and preload them into each layer at once to perform inference for each corresponding inference request.

[0057] Based on step S300, from the above expert activation mode In the process, the top MoE layers with the highest activation probability values ​​are obtained sequentially. ( (A natural number not less than 1) number of experts, which are preloaded into each layer at once for subsequent reasoning.

[0058] In summary, this embodiment leverages the expert activation similarity exhibited by adjacent tokens generated in the early decoding stage within the same MoE layer. Combined with expert score design, it simplifies expert selection at the input prompt word granularity, resolving the resource overhead issues associated with complex expert selection and prediction at the traditional token granularity level. Furthermore, by employing one-time expert prediction and prefetching, it effectively improves inference efficiency while ensuring inference accuracy, achieving a performance balance between the two.

[0059] Example 2:

[0060] This embodiment also provides a processing apparatus, including:

[0061] The configuration module is used to configure and edit the protocol file corresponding to the implementation of the MoE large language model inference acceleration method on resource-constrained devices, wherein the protocol file is binary format data. Specifically, the steps for configuring and editing the protocol file are as described in Embodiment 1.

[0062] The parsing module is used to parse the protocol file into JSON format data. Specifically, the steps for parsing the protocol file into JSON format data are as described in Example 1.

[0063] Example 3:

[0064] Those skilled in the art will understand that all or part of the features / steps of the above-described method embodiments can be implemented by methods, data processing systems, or computer programs. These features may be implemented without hardware, entirely in software, or in a combination of hardware and software. The aforementioned computer program may be stored in one or more computer-readable storage media. When the computer program is executed (e.g., by a processor), it performs the steps of the above-described embodiment of a method for accelerating inference of a MoE large language model on a resource-constrained device.

[0065] The aforementioned storage media capable of storing program code include: static disks, solid-state drives, random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), optical storage devices, magnetic storage devices, flash memory, magnetic disks or optical disks, and / or combinations of the above devices, that is, they can be implemented by any type of volatile or non-volatile storage devices or combinations thereof.

[0066] Example 4:

[0067] like Figure 2 As shown, the present invention also provides an electronic device, comprising:

[0068] One or more processors;

[0069] A memory for storing one or more computer programs, and one or more processors for executing the one or more computer programs stored in the memory to cause the one or more processors to perform features / steps of the inference acceleration method for the MoE large language model on a resource-constrained device as described in Embodiment 1.

[0070] Furthermore, at the hardware level, the device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other necessary hardware. One or more embodiments of this application can be implemented in software, such as by the processor reading the corresponding computer program from the non-volatile memory into memory and then running it.

[0071] Of course, in addition to the software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software. That is to say, the execution subject of the above processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0072] It should be understood that the above embodiments are only special cases and do not indicate that the present invention is implemented in such a way.

[0073] Although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0074] The above description is merely a preferred embodiment of the present invention. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for accelerating inference of a MoE large language model on resource-constrained devices, characterized in that, Includes the following steps: The expert activation mode corresponding to the prompt word data is obtained by reasoning through the MoE large language model and calculating the expert score in the token generation process. The prompt word data and its corresponding expert activation mode are used as binary data pairs to form a dataset, and the dataset is used to train the expert activation prediction model. The MoE large language model is used to perform initial inference for each inference request, and the trained expert activation prediction model is used to predict the expert activation mode of each inference request at each layer of the MoE large language model. Based on the predicted activation probability value, select the corresponding expert from each layer of the MoE large language model corresponding to each inference request from the predicted expert activation mode, and preload them into each layer at once to perform inference for each corresponding inference request. The k-th expert score of the l-th layer of the MoE large language model corresponding to the i-th prompt word data. It can be expressed by the following formula: ； in, This indicates that it corresponds to an expert. Routing weights and These represent the number of experts at each layer of the MoE large language model and the number of layers in the MoE large language model, respectively. Indicates an indicator function, if Used for inference generation If the value is 1, then the value is 1; otherwise, the value is 0. The length of the response output obtained by reasoning from the i-th prompt word data; For the i-th prompt word data, the expert scores of the experts tracked by the indicator function at different layers of the MoE large language model together constitute the expert activation mode corresponding to the i-th prompt word data.

2. The method for accelerating inference of a MoE large language model on resource-constrained devices according to claim 1, characterized in that, The expert activation prediction model is independent of the MoE large language model and is used to predict the experts that need to be activated in all layers of the MoE large language model at once according to the corresponding inference request. The expert activation prediction model takes a cue word as input and outputs the expert activation pattern corresponding to the cue word, representing the importance probability of all experts being activated across all layers.

3. The method for accelerating inference of a MoE large language model on resource-constrained devices according to claim 1, characterized in that, The expert activation prediction model includes a pre-trained language model encoding module and a multilayer perceptron prediction module, wherein... The pre-trained language model encoding module is used to capture semantic features in the input prompt words and generate prompt word embeddings; The multilayer perceptron prediction module is used to learn the mapping relationship between cue word embeddings and expert activation modes.

4. The method for accelerating inference of a MoE large language model on resource-constrained devices according to claim 1, characterized in that, Before performing initial inference for each inference request using the MoE large language model, the following steps are also included: Non-expert parameters are stored in GPU memory, while expert parameters are loaded into CPU memory. Simultaneously, the trained expert activation prediction model is loaded into CPU memory for the MoE large language model.

5. The method for accelerating inference of a MoE large language model on resource-constrained devices according to claim 1, characterized in that, When training the expert activation prediction model using the dataset, the predicted expert activation pattern corresponding to each prompt word data is fitted to its true expert activation pattern by minimizing the mixed loss function composed of cross-entropy loss and Huber loss, thus obtaining the trained expert activation prediction model.

6. A processing apparatus, characterized in that, include: The configuration module is used to configure and edit the protocol file corresponding to the inference acceleration method of MoE large language model on resource-constrained devices according to any one of claims 1-5, wherein the protocol file is binary format data; The parsing module is used to parse the protocol file into JSON format data.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed, implements the inference acceleration method for a MoE large language model on a resource-constrained device according to any one of claims 1-5.

8. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more computer programs, and one or more processors for executing the one or more computer programs stored in the memory to cause the one or more processors to perform the inference acceleration method for the MoE large language model on a resource-constrained device as described in any one of claims 1-5.

Images