Unloading method and device for reasoning task of sparse hybrid expert large language model

Abstract

The invention belongs to the field of artificial intelligence, and particularly relates to an unloading method and device for a sparse hybrid expert large language model reasoning task. The method comprises the following steps: predicting selected experts and corresponding active neurons of the sparse hybrid expert large language model in a reasoning task; the active neurons of the selected expert are loaded at a computing device to perform the reasoning task. According to the method and the device, the reasoning delay when the sparse hybrid expert large language model is unloaded on the consumer-level GPU when executing the reasoning task is reduced.

Description

Technical Field

[0001] The present disclosure belongs to the field of artificial intelligence, and specifically relates to a method and device for offloading a sparse mixed expert large language model reasoning task. Background Art

[0002] Sparse mixture of expert large language models are becoming more and more common in AI applications, such as chatbots and code generation. Sparse mixture of experts is a neural network design whose core idea is to activate only some of the model parameters to participate in the calculation (i.e., experts) based on a given input. Sparse mixture of expert large language models can scale their model parameters with almost no increase in computational cost. Compared with dense large language models, deploying sparse mixture of expert large language models on consumer-grade GPUs faces significant challenges because sparse mixture of expert large language models are usually equipped with more model parameters than their dense counterparts, far exceeding the memory capacity of consumer-grade devices.

[0003] Offloading technology is one of the main methods for deploying sparse mixed expert large language models on resource-limited devices. Offloading technology stores experts in CPU DRAM to meet the memory limitations of consumer-grade GPUs. However, this will lead to unacceptable inference latency. For example, the sparse mixed expert large language model Mixtral8×7B, each of which has 8 experts in the expert layer, requires 94GB of memory in half-precision representation. The offloading method based on Mixtral-Offload stores the experts in CPU DRAM and loads the corresponding experts to GPU memory when needed. Although this makes it possible to perform inference on the Mixtral 8×7B model in a GPU with 24GB of memory, the expert loading operation significantly increases the inference latency, with an inference latency of about 3.9 seconds per word, which is significantly slower than the human reading speed of 0.2 to 0.3 seconds; the Fiddler-based offloading method stores the experts of the Mixtral 8×7B model in CPU DRAM, and all expert-related calculations are performed on the CPU side during inference. Although this avoids a large number of loading operations between the CPU and the GPU, since the computing power of the CPU is significantly lower than that of the GPU, the expert-related calculations also significantly increase the inference latency. How to avoid the significant inference delay caused by offloading technology is the core problem that needs to be solved. Summary of the invention

[0004] The disclosed embodiment proposes an offloading solution for sparse mixed expert large language model reasoning tasks, which reduces the problem of significant reasoning delay in existing offloading solutions on consumer-grade GPUs.

[0005] A first aspect of an embodiment of the present disclosure provides a method for offloading a sparse mixed expert large language model reasoning task, comprising:

[0006] Predicting the selected experts and corresponding active neurons of the sparse mixture of experts large language model in the reasoning task;

[0007] The active neurons of the selected expert are loaded on a computing device to perform the reasoning task.

[0008] In some embodiments, the computing device includes a CPU and a GPU, the sparse mixed expert large language model includes multiple expert layers, each of the expert layers includes multiple experts, and the method further includes:

[0009] When the expert layer at the i-th layer performs the reasoning task, the active neurons of the selected expert at the expert layer at the i+1-th layer are synchronously pre-fetched on the GPU side, where i is a positive integer.

[0010] In some embodiments, pre-fetching the active neurons of the selected expert in the expert layer of the i+1th layer on the GPU side comprises:

[0011] Synchronously predicting the selected expert of the expert layer of the i+1th layer based on the input of the expert layer of the ith layer;

[0012] synchronously predicting the active neurons of the expert;

[0013] Synchronously pre-fetch the active neurons to the GPU.

[0014] The method of predicting the selected expert of the expert layer of the i+1th layer based on the input of the expert layer of the ith layer comprises:

[0015] The hidden state output by the i-th attention layer of the sparse mixed expert large language model is used as the input of the gating unit of the i+1-th expert layer, and the selected expert predicted by the i+1-th expert layer is output, wherein the sparse mixed expert large language model includes a layered structure, each layer of the layered structure is composed of a series-connected attention layer and an expert layer, the hidden state output by the attention layer is used as the input of the expert layer, and the expert layer includes a gating unit for activating the experts involved in the calculation.

[0016] In some embodiments, predicting the active neurons of the expert comprises:

[0017] The active neurons of the expert are predicted based on the weights of the expert.

[0018] In some embodiments, predicting the active neurons of the expert based on the expert's weights comprises:

[0019] Get the weight matrix of the expert,

[0020] An activation function is used to activate a matrix multiplication calculation result of the weight matrix and the input hidden state, and a neuron corresponding to an activation result whose absolute value exceeds a preset threshold is taken as the active neuron.

[0021] In some embodiments, the GPU side includes an expert cache for caching experts and a neuron cache for caching the active neurons of each of the experts, and loading the active neurons of the selected expert on the computing device includes:

[0022] If the selected expert is in the expert cache, and the active neuron is in the neuron cache corresponding to the expert cache, reading the active neuron from the neuron cache to perform the reasoning task;

[0023] If the selected expert is in the expert cache and the active neuron is not in the neuron cache corresponding to the expert cache, cache the active neuron in the neuron cache, and read the active neuron from the neuron cache to perform the reasoning task;

[0024] If the selected expert is not in the expert cache, the selected expert is cached in the expert cache, and the active neurons of the expert are cached in the neuron cache corresponding to the expert cache, and the active neurons are read from the neuron cache to perform the reasoning task.

[0025] In some embodiments, caching the selected expert in the expert cache comprises:

[0026] Loading the selected expert into the expert cache based on an LRU strategy;

[0027] Cache the active neurons of the expert in the neuron cache corresponding to the expert cache, including:

[0028] When the selected expert is not in the expert cache, loading all active neurons into the neuron cache corresponding to the expert cache;

[0029] When the selected expert is in the expert cache, all active neurons that are not in the neuron cache corresponding to the expert cache are loaded into the neuron cache.

[0030] In some embodiments, loading the active neurons comprises:

[0031] Compressing the active neurons into a compressed matrix at the CPU end;

[0032] The compressed matrix is ​​loaded from the CPU side to the GPU side and decompressed.

[0033] A second aspect of the embodiments of the present disclosure provides a device for offloading a sparse mixed expert large language model reasoning task, comprising:

[0034] A prediction module, used to predict the selected experts and corresponding active neurons of the sparse mixture of experts large language model in the reasoning task;

[0035] An execution module is used to load the selected active neurons of the expert on a computing device to execute the reasoning task.

[0036] In summary, the methods, devices and equipment for offloading reasoning tasks of sparse hybrid expert large language models provided in the embodiments of the present disclosure reduce the reasoning delay when the sparse hybrid expert large language model performs reasoning tasks and is unloaded on a consumer-grade GPU. In particular, by utilizing the dual sparsity of the expert and neuron levels to optimize the weight loading process, the problem that the existing methods cannot strike a balance between reasoning delay and accuracy is solved; through the coarse-grained to fine-grained prediction and pre-fetch mechanism, the required weights are accurately predicted without increasing the additional memory requirements, thereby achieving the overlap of weight loading and calculation to hide the delay, solving the execution sequentiality problem in the sparse hybrid expert large language model; through the coarse-grained to fine-grained caching strategy, the need for repeated weight loading is minimized, solving the problem of inefficient GPU memory utilization when offloading reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0038] Figure 1 is a schematic diagram of a smart device to which the present disclosure is applicable;

[0039] Figure 2 is a schematic diagram of an existing unloading solution and an unloading solution proposed in the present disclosure;

[0040] Figure 3 is a flowchart of a method for offloading a sparse mixed expert large language model reasoning task according to some embodiments of the present disclosure;

[0041] Figure 4 is a schematic diagram of a sparse mixed expert large language model expert selection and neuron sparsity predictor according to some embodiments of the present disclosure;

[0042] Figure 5 is a schematic diagram of a sparse mixed expert large language model expert caching strategy from coarse-grained to fine-grained according to some embodiments of the present disclosure;

[0043] Figure 6 is the speedup ratio of the offloading method proposed in the present disclosure to the baseline end-to-end inference latency;

[0044] Figure 7 is a comparison table of the accuracy of the offloading method proposed in the present disclosure against the baseline model;

[0045] Figure 8 is a schematic diagram of an offloading device for a sparse mixed expert large language model reasoning task according to some embodiments of the present disclosure; DETAILED DESCRIPTION

[0046] In the detailed description below, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be implemented without these details. It should be understood that the use of the terms "system," "device," "unit," and / or "module" in the present disclosure is a method for distinguishing different parts, elements, parts, or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if other expressions can achieve the same purpose.

[0047] It should be understood that when a device, unit or module is referred to as being "on," "connected to," or "coupled to" another device, unit or module, it may be directly on, connected to, coupled to, or communicating with other devices, units or modules, or there may be intermediate devices, units or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used in this disclosure includes any and all combinations of one or more of the relevant listed items.

[0048] The terms used in this disclosure are only for describing specific embodiments and are not intended to limit the scope of this disclosure. As shown in the specification and claims of this disclosure, unless the context clearly indicates an exception, the words "one", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.

[0049] These and other features and characteristics, methods of operation, functions of related elements of the structure, combinations of parts, and economies of manufacture of the present disclosure may be better understood with reference to the following description and drawings, which form a part of the specification. However, it is to be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present disclosure. It is to be understood that the drawings are not drawn to scale.

[0050] Various structural diagrams are used in this disclosure to illustrate various variations of the embodiments of the present disclosure. It should be understood that the above or below structures are not intended to limit the present disclosure. The scope of protection of the present disclosure shall be subject to the claims.

[0051] Figure 1 It is a schematic diagram of a smart device to which the present disclosure is applicable. Figure 1 The device shown includes a loading module connected to the GPU and the CPU data, and the loading module is used to load the sparse mixed expert large language model predicting the selected expert and the corresponding active neurons from the device memory by the CPU to the GPU memory to perform the reasoning task.

[0052] The sparse mixed expert large language model can activate only part of the model parameters to participate in the calculation according to the given input. The sparse mixed expert large language model generally replaces the FFN layer of the conventional dense large language model with a mixed expert layer. Each mixed expert layer contains a certain number of experts and a gating unit to decide which experts to activate to participate in the calculation. The CPU (Central Processing Unit) is used for general computing and can directly manage the device memory (DRAM). The device memory generally has a large capacity. The GPU (Graphics Processing Unit) contains a large number of computing cores and is good at processing parallel computing tasks. Neural network calculations are generally handled by the GPU. The GPU has a built-in GPU memory of a smaller capacity. The loading module is a hardware or software module embedded in the smart device.

[0053] Sparse mixed expert large language models are usually equipped with more model parameters than their dense counterparts, far exceeding the memory capacity of consumer-grade GPUs. For example, the Mixtral 8×7B, the best performing sparse mixed expert large language model with the same model parameter volume, has 8 experts in each mixed expert layer and requires 94GB of memory in half-precision representation. However, NVIDIA RTX 3090Ti only has 24GB of memory.

[0054] Offload technology stores experts in CPU DRAM, thereby meeting the memory limitations of consumer-grade GPUs. However, this will result in unacceptable inference latency. Take Mixtral 8×7B as an example: Mixtral-Offload stores experts in CPU DRAM and loads the corresponding experts into GPU memory when needed. Although this makes it possible to perform inference on the Mixtral 8×7B model in a GPU with 24GB of memory, the expert loading operation significantly increases the inference latency, with an inference latency of about 3.9 seconds per word, which is significantly slower than the human reading speed of 0.2 to 0.3 seconds. Fiddler stores the experts of the Mixtral 8×7B model in CPU DRAM, and all expert-related calculations are performed on the CPU side during inference. Although this avoids a large number of loading operations between the CPU and GPU, since the computing power of the CPU is significantly lower than that of the GPU, the expert-related calculations also significantly increase the inference latency.

[0055] In order to solve the problem that the existing offloading solution of sparse mixed expert large language model reasoning tasks has significant inference delay on consumer-grade GPUs, this disclosure proposes an offloading solution of sparse mixed expert large language model reasoning tasks based on double-layer offloading. Figure 2 As shown, compared with the existing solution of densely loading all neurons into GPU memory, the present disclosure performs sparse loading based on the sparsity of neurons in two dimensions, coarse-grained and fine-grained. Coarse-grained refers to the expert dimension, and fine-grained refers to the neuron dimension, where a neuron refers to a row or column of the weight matrix. Specifically, the required experts and neurons are dynamically predicted during reasoning, and pre-fetched weights are used to improve parallelism. Only some neurons in the experts are loaded for calculation, thereby reducing the weight loading amount and improving throughput. In addition, the cache is maintained on the GPU memory from the two dimensions of experts and neurons to minimize the repeated loading of weights and further improve the reasoning efficiency.

[0056] Figure 3 is a flowchart of a method for offloading a sparse mixed expert large language model reasoning task according to some embodiments of the present disclosure. In some embodiments, the method for offloading a sparse mixed expert large language model reasoning task is Figure 1 The unloading module shown is executed, and the unloading method of the sparse mixed expert large language model reasoning task includes the following steps:

[0057] S310, predicting the experts and corresponding active neurons selected by the sparse mixed expert large language model in the reasoning task.

[0058] Prefetching is a method of moving the required data from slow storage to fast storage in advance to reduce the waiting time for future use of the data. In large language model reasoning, it means moving the experts or neurons that will be needed in the future to the GPU in advance. This movement process is performed in parallel with the calculation process in the GPU. The prefetching method masks part of the calculation delay, thereby reducing the reasoning delay.

[0059] However, in the sparse mixed expert large language model, the choice of experts and the sparsity of neurons are dynamic. Only when we execute to the corresponding mixed expert layer can we get the exact expert choice and active neurons. This sequential execution makes it impossible to pre-fetch weights. In order to know the sparsity of neurons within the expert in advance, we need to make corresponding predictions. Previous work adds and trains a predictor for each layer, but if applied to the sparse mixed expert large language model, it will result in unacceptable additional memory overhead, especially in our scenario with memory-constrained consumer-grade GPUs.

[0060] In order to save memory, this paper uses the weight of the model itself to predict the expert's choice and the sparsity of neurons. We add a predictor after the i-th attention layer to predict the i+1-th mixed expert layer. The predictor structure is as follows: Figure 4 As shown. In the i-th predictor, the input needs to pass through the gated unit of the i+1th layer to obtain the prediction of the i+1th layer expert. Then the weight layer of the prediction expert is pre-fetched to the GPU, the input and the weight layer are calculated, and then the calculation result is passed through the activation function. The present disclosure selects the corresponding neurons whose absolute value exceeds the threshold through the result of the activation function to obtain the prediction of the active neurons.

[0061] By predicting sparsity in advance, we can pre-fetch the required weights, mask some of the computational latency, and reduce the inference latency. During the entire pre-fetch process, model inference is performed normally. This disclosure uses CUDA streams to achieve parallel weight loading and computation. The pre-fetch of the i+1 layer will start executing at the same time as the hybrid expert layer of the i layer, and will be globally synchronized until the predictor of the i+2 layer is executed, until the pre-fetch process of the i+1 layer is completed.

[0062] S320: Load the selected active neurons of the expert on a computing device to perform the reasoning task.

[0063] After determining the experts and neurons needed for each reasoning, how to efficiently use the GPU memory to reduce the loading of experts and neurons is an issue that needs further consideration. Expert cache can effectively reduce the latency of reasoning sparse hybrid expert large language models using the offloading method. Sparse hybrid expert large language models often reuse experts in multiple iterations of the reasoning process, so caching these reused experts in a memory location closer to the GPU, such as GPU HBM, can avoid expensive retrieval from CPU DRAM. Previous work has shown that when the maintenance strategy of the expert cache uses the simplest LRU, a satisfactory hit rate can still be achieved. In some embodiments of the present disclosure, a fixed storage space will be opened up in the GPU memory before the reasoning starts, which is used to cache a fixed number of experts and avoid exceeding the GPU memory limit. During reasoning, the cached experts are maintained using the LRU strategy. If an expert is selected and has been stored in the cache, data loading can be avoided, thereby improving efficiency.

[0064] However, since the double-layer sparse loading we introduced when loading weights is not a complete loading of the expert, even if the expert cache stores the expert selected by the current token, we cannot guarantee that the reasoning of this token is correct, because different inputs will produce different neuron sparsity. It is very likely that the neurons that are active during this token reasoning were judged as inactive last time and were not loaded into the GPU expert cache. Therefore, we implemented a coarse-grained to fine-grained caching strategy. Figure 5 As shown in Figure 1, we maintain a neuron cache for each expert in the expert cache. For experts already stored in the expert cache, we only load neurons that affect the results and are not in the neuron cache, and update the expert's neuron cache by taking the union of the cached neurons and the neurons active during the current reasoning. For experts who have newly entered the expert cache, we directly perform sparse loading of neurons and set the expert's neuron cache to the currently loaded neurons. In summary, the hierarchical caching of experts and neurons from coarse-grained to fine-grained caching significantly reduces the weight loading delay caused by unloading.

[0065] Finally, the active neurons in the expert are compressed into a matrix on the CPU side, then loaded into the GPU side, and finally decompressed on the GPU side. Double-layer sparse loading significantly reduces the amount of weights loaded and reduces loading latency.

[0066] An embodiment of the present disclosure conducts an end-to-end experiment based on the edge device reasoning scenario based on the method described in S310-S320.

[0067] In the experiments, the input length is 64 and 128, while the output length varies from 8 to 512. Sampling is performed on C4 and Wikipedia datasets, which are high-quality text datasets covering a wide range of domains. The baseline implementation uses expert offloading. Specifically, the model weights are quantized to INT4 and the intermediate activations are stored as FP16. The non-mixed expert layer weights and activations of the sparse mixed expert large language model are stored on the GPU, while the mixed expert layer weights are stored in the CPU's DRAM. This is a common offloading strategy for sparse mixed expert large language models in previous work. Figure 6 A comparison of end-to-end generation speedups at different input and output lengths is shown. For Mixtral 8×7B, the average speedups are 1.92x and 1.99x for input lengths of 64 and 128, respectively, while for DeepSeekMoE 16B, the average speedups are 2.04x and 2.09x for the same input lengths, respectively. The speedup effect of the method used in the present disclosure increases with increasing generation length. This enhancement occurs because the number of generated tokens increases, the cache hit rate increases through the coarse-to-fine-grained cache system, and the weight loading latency is reduced.

[0068] The method used in the present disclosure predicts the usage of experts and the sparsity of neurons, and ignores some neurons during the reasoning process, which has a certain impact on the accuracy of the sparse mixed expert large language model. Figure 7 The accuracy with and without our method is compared. Experiments show that our method maintains competitive accuracy across different models and various downstream tasks. The accuracy of sparse mixture of experts large language models may fluctuate due to expert selection errors and omission of some actually active neurons. However, sometimes this even improves the accuracy.

[0069] Figure 8 is a schematic diagram of an offloading device for a sparse mixed expert large language model reasoning task according to some embodiments of the present disclosure. Figure 8 As shown, the sparse mixed expert large language model reasoning task offloading device 800 includes a prediction module 810 and an execution module 820. In some embodiments of the present disclosure, the sparse mixed expert large language model reasoning task offloading function is Figure 1 The uninstall module shown is executed.

[0070] A prediction module 810, configured to predict the selected experts and corresponding active neurons of the sparse mixture of experts large language model in the reasoning task;

[0071] The execution module 820 is used to load the selected active neurons of the expert on a computing device to execute the reasoning task.

[0072] In summary, the methods, devices and equipment for offloading reasoning tasks of sparse hybrid expert large language models provided in the embodiments of the present disclosure reduce the reasoning delay when the sparse hybrid expert large language model performs reasoning tasks and is unloaded on a consumer-grade GPU. In particular, by utilizing the dual sparsity of the expert and neuron levels to optimize the weight loading process, the problem that the existing methods cannot strike a balance between reasoning delay and accuracy is solved; through the coarse-grained to fine-grained prediction and pre-fetch mechanism, the required weights are accurately predicted without increasing the additional memory requirements, thereby achieving the overlap of weight loading and calculation to hide the delay, solving the execution sequentiality problem in the sparse hybrid expert large language model; through the coarse-grained to fine-grained caching strategy, the need for repeated weight loading is minimized, solving the problem of inefficient GPU memory utilization when offloading reasoning.

[0073] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding descriptions in the aforementioned device embodiments and will not be repeated here.

[0074] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of operating systems and application programs on computer systems, it will be appreciated by those skilled in the art that other implementations may also be performed in conjunction with other types of program modules. In general, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. It will be appreciated by those skilled in the art that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc., and may also be used in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0075] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0076] It should be understood that the above specific embodiments of the present disclosure are only used to illustrate or explain the principles of the present disclosure, and do not constitute a limitation of the present disclosure. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present disclosure should be included in the protection scope of the present disclosure. In addition, the claims attached to the present disclosure are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries.

Claims

1. A method for offloading sparse mixed-expert large language model reasoning tasks, characterized in that: include: Predicting the selected experts and corresponding active neurons of the sparse mixture of experts large language model in the reasoning task; The active neurons of the selected expert are loaded on a computing device to perform the reasoning task.

2. According to the method of claim 1, the computing device includes a CPU end and a GPU end, the sparse mixed expert large language model includes multiple expert layers, each of the expert layers includes multiple experts, characterized in that: The method further comprises: When the expert layer at the i-th layer performs the reasoning task, the active neurons of the selected expert at the expert layer at the i+1-th layer are synchronously pre-fetched on the GPU side, where i is a positive integer.

3. The method according to claim 2, characterized in that: The pre-fetching the active neurons of the expert selected in the expert layer of the i+1th layer on the GPU side comprises: Synchronously predicting the selected expert of the expert layer of the i+1th layer based on the input of the expert layer of the ith layer; synchronously predicting the active neurons of the expert; Synchronously pre-fetch the active neurons to the GPU.

4. The method according to claim 3, characterized in that: The method of predicting the selected expert of the expert layer of the i+1th layer based on the input of the expert layer of the ith layer comprises: The hidden state output by the i-th attention layer of the sparse mixed expert large language model is used as the input of the gating unit of the i+1-th expert layer, and the selected expert predicted by the i+1-th expert layer is output, wherein the sparse mixed expert large language model includes a layered structure, each layer of the layered structure is composed of a series-connected attention layer and an expert layer, the hidden state output by the attention layer is used as the input of the expert layer, and the expert layer includes a gating unit for activating the experts involved in the calculation.

5. The method according to claim 4, characterized in that: The predicting of the active neurons of the expert comprises: The active neurons of the expert are predicted based on the weights of the expert.

6. The method according to claim 5, characterized in that: The predicting the active neurons of the expert based on the expert's weights comprises: Get the weight matrix of the expert, An activation function is used to activate a matrix multiplication calculation result of the weight matrix and the input hidden state, and a neuron corresponding to an activation result whose absolute value exceeds a preset threshold is taken as the active neuron.

7. The method according to claim 2, characterized in that: The GPU side includes an expert cache for caching experts and a neuron cache for caching the active neurons of each of the experts, and the loading of the active neurons of the selected expert on the computing device includes: If the selected expert is in the expert cache, and the active neuron is in the neuron cache corresponding to the expert cache, reading the active neuron from the neuron cache to perform the reasoning task; If the selected expert is in the expert cache and the active neuron is not in the neuron cache corresponding to the expert cache, cache the active neuron in the neuron cache, and read the active neuron from the neuron cache to perform the reasoning task; If the selected expert is not in the expert cache, the selected expert is cached in the expert cache, and the active neurons of the expert are cached in the neuron cache corresponding to the expert cache, and the active neurons are read from the neuron cache to perform the reasoning task.

8. The method according to claim 7, characterized in that: The expert cache to be selected includes: Loading the selected expert into the expert cache based on an LRU strategy; Cache the active neurons of the expert in the neuron cache corresponding to the expert cache, including: When the selected expert is not in the expert cache, loading all active neurons into the neuron cache corresponding to the expert cache; When the selected expert is in the expert cache, all active neurons that are not in the neuron cache corresponding to the expert cache are loaded into the neuron cache.

9. The method according to claim 8, characterized in that: Loading the active neuron includes: Compressing the active neurons into a compressed matrix at the CPU end; The compressed matrix is ​​loaded from the CPU side to the GPU side and decompressed.

10. A device for offloading sparse mixed expert large language model reasoning tasks, characterized in that: include: A prediction module, used to predict the selected experts and corresponding active neurons of the sparse mixture of experts large language model in the reasoning task; An execution module is used to load the selected active neurons of the expert on a computing device to execute the reasoning task.

Images