A method and device for offloading inference tasks of a sparse mixed expert large language model

Abstract

The present disclosure belongs to the field of artificial intelligence, and particularly relates to a method and device for unloading a sparse mixed expert large language model inference task. The method comprises: predicting experts selected by the sparse mixed expert large language model in an inference task and corresponding active neurons; and loading the active neurons of the selected experts on a computing device to execute the inference task. The present disclosure reduces the inference delay of the sparse mixed expert large language model when unloading on a consumer-grade GPU when executing an inference task.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the field of artificial intelligence, and particularly relates to a method and device for offloading inference tasks of a sparse hybrid expert large language model. BACKGROUND

[0002] The application of sparse hybrid expert large language models in the field of AI is becoming more and more common, such as chat robots and code generation. Sparse hybrid experts are a kind of neural network design, whose core idea is to activate only part of the model parameters to participate in the calculation (i.e. experts) according to the given input. Sparse hybrid expert large language models can expand their model parameter scale without significantly increasing the computing cost. Compared with dense large language models, deploying sparse hybrid expert large language models on consumer-grade GPUs faces major challenges because sparse hybrid expert large language models are usually equipped with more model parameters than their dense counterparts, far exceeding the memory capacity of consumer-grade devices.

[0003] Offloading technology is one of the main methods for deploying sparse hybrid expert large language models on resource-limited devices. Offloading technology stores experts in CPU DRAM, thereby meeting the memory limit of consumer-grade GPUs. However, this results in unacceptable inference delay. For example, Mixtral8x7B, a sparse hybrid expert large language model, has 8 experts per expert layer and requires 94 GB of memory in half-precision representation. The offloading method based on Mixtral-Offload stores experts in CPU DRAM and loads the corresponding experts into GPU memory when needed. Although this makes it possible for the Mixtral 8x7B model to perform inference on a GPU with 24 GB of memory, the expert loading operation significantly increases the inference delay, with an inference delay of about 3.9 seconds per word, which is significantly slower than the human reading speed of 0.2 to 0.3 seconds; the offloading method based on Fiddler stores the experts of the Mixtral 8x7B model in CPU DRAM, and all computations related to the experts are performed on the CPU side during inference. Although this avoids a large number of loading operations between the CPU and the GPU, the computation related to the experts also significantly increases the inference delay because the computing power of the CPU is significantly lower than that of the GPU. How to avoid the significant inference delay caused by offloading technology is a core problem that needs to be solved. SUMMARY

[0004] The embodiments of the present disclosure propose an offloading scheme for inference tasks of a sparse hybrid expert large language model, which reduces the problem of significant inference delay on consumer-grade GPUs in existing offloading schemes.

[0005] A first aspect of the embodiments of the present disclosure provides an offloading method for inference tasks of a sparse hybrid expert large language model, comprising:

[0006] predicting experts and corresponding active neurons of the sparse hybrid expert large language model selected in an inference task;

[0007] loading the active neurons of the selected experts by the computing device to perform the inference task.

[0008] In some embodiments, the computing device comprises a CPU side and a GPU side, the sparse hybrid expert large language model comprises a plurality of expert layers, each of the expert layers comprises a plurality of experts, and the method further comprises:

[0009] synchronously pre-fetching the active neurons of the selected experts of an (i+1)th expert layer on the GPU side when an ith expert layer performs the inference task, wherein i is a positive integer.

[0010] In some embodiments, the pre-fetching the active neurons of the selected experts of the (i+1)th expert layer on the GPU side comprises:

[0011] synchronously predicting the selected experts of the (i+1)th expert layer based on inputs of the ith expert layer;

[0012] synchronously predicting the active neurons of the experts;

[0013] synchronously pre-fetching the active neurons to the GPU side.

[0014] The predicting the selected experts of the (i+1)th expert layer based on inputs of the ith expert layer comprises:

[0015] outputting a hidden state of an ith attention layer of the sparse hybrid expert large language model as an input of a gating unit of an (i+1)th expert layer, and outputting the selected experts predicted by the (i+1)th expert layer, wherein the sparse hybrid expert large language model comprises a layered structure, each of the layered structures is composed of an attention layer and an expert layer connected in series, a hidden state output by the attention layer is used as an input of the expert layer, and the expert layer comprises a gating unit for activating the experts participating in calculation.

[0016] In some embodiments, the predicting the active neurons of the experts comprises:

[0017] predicting the active neurons of the experts based on weights of the experts.

[0018] In some embodiments, the predicting the active neurons of the experts based on weights of the experts comprises:

[0019] obtaining a weight matrix of the experts,

[0020] activating the weight matrix and the matrix multiplication result of the hidden state of the input with an activation function, and taking a neuron corresponding to an activation result exceeding a preset threshold as the active neuron.

[0021] In some embodiments, the GPU side includes an expert cache for caching experts and a neuron cache for caching the active neurons of each of the experts, and the loading of the active neurons of the selected expert by the computing device includes:

[0022] if the selected expert is in the expert cache and the active neurons are in the neuron cache corresponding to the expert cache, reading the active neurons from the neuron cache to perform the inference task;

[0023] if the selected expert is in the expert cache and the active neurons are not in the neuron cache corresponding to the expert cache, caching the active neurons in the neuron cache and reading the active neurons from the neuron cache to perform the inference task;

[0024] if the selected expert is not in the expert cache, caching the selected expert in the expert cache and caching the active neurons of the expert in the neuron cache corresponding to the expert cache, reading the active neurons from the neuron cache to perform the inference task.

[0025] In some embodiments, the caching of the selected expert in the expert cache includes:

[0026] loading the selected expert into the expert cache based on an LRU policy;

[0027] the caching of the active neurons of the expert in the neuron cache corresponding to the expert cache includes:

[0028] when the selected expert is not in the expert cache, loading all the active neurons into the neuron cache corresponding to the expert cache;

[0029] when the selected expert is in the expert cache, loading all the active neurons not in the neuron cache corresponding to the expert cache into the neuron cache.

[0030] In some embodiments, the loading of the active neurons includes:

[0031] compressing the active neurons into a compressed matrix at the CPU side;

[0032] loading and decompressing the compressed matrix from the CPU side to the GPU side.

[0033] A second aspect of the embodiments of the present disclosure provides a device for offloading a sparse hybrid expert large language model inference task, comprising:

[0034] a prediction module configured to predict an expert of the sparse hybrid expert large language model selected in an inference task and a corresponding active neuron;

[0035] an execution module configured to load the active neuron of the selected expert on a computing device to perform the inference task.

[0036] In summary, the method, device and equipment for offloading a sparse hybrid expert large language model inference task provided by the embodiments of the present disclosure reduce the inference delay of the sparse hybrid expert large language model when offloaded on a consumer-grade GPU when performing an inference task. In particular, the double sparsity of experts and neurons is utilized to optimize the weight loading process, solving the problem that existing methods cannot balance between inference delay and accuracy. Through the coarse-to-fine prediction and prefetching mechanism, the required weights are accurately predicted without increasing the additional memory requirement, so as to overlap the weight loading and calculation to hide the delay, solving the execution sequence problem in the sparse hybrid expert large language model. Through the coarse-to-fine cache strategy, the demand for repeated loading of weights is maximally reduced, solving the problem of low efficiency of GPU memory utilization when offloading the inference. BRIEF DESCRIPTION OF DRAWINGS

[0037] The features and advantages of the present disclosure will be more clearly understood through reference to the following drawings, which are presented as illustrative and should not be construed as limiting the present disclosure, in which:

[0038] Figure 1 is a schematic diagram of an intelligent device to which the present disclosure is applicable;

[0039] Figure 2 is a schematic diagram of an existing offloading scheme and the offloading scheme proposed by the present disclosure;

[0040] Figure 3 is a flowchart of a method for offloading a sparse hybrid expert large language model inference task according to some embodiments of the present disclosure;

[0041] Figure 4 is a schematic diagram of a sparse hybrid expert large language model expert selection and neuron sparsity predictor according to some embodiments of the present disclosure;

[0042] Figure 5 is a schematic diagram of a sparse hybrid expert large language model expert from coarse-to-fine cache strategy according to some embodiments of the present disclosure;

[0043] Figure 6 is a baseline end-to-end inference latency acceleration ratio of the offloading method proposed by the present disclosure;

[0044] Figure 7 is a baseline model accuracy comparison table of the offloading method proposed by the present disclosure;

[0045] Figure 8 is a schematic diagram of a sparse mixed expert large language model inference task offloading device according to some embodiments of the present disclosure; DETAILED DESCRIPTION

[0046] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be practiced without these details. It will be understood that the use of "system", "device", "unit", and / or "module" terminology in the present disclosure is used as a method of distinguishing different components, elements, parts, or assemblies in sequential arrangement. However, these terms can be replaced by other expressions as long as the same purpose is achieved.

[0047] It should be understood that when a device, unit, or module is referred to as being "on", "connected to", or "coupled to" another device, unit, or module, it can be directly on, connected or coupled to, or in communication with other devices, units, or modules, or there can be intervening devices, units, or modules, unless the context clearly indicates otherwise. For example, the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0048] The terminology used by the present disclosure is only for the purpose of describing particular embodiments and is not intended to limit the scope of the present disclosure. As used in the specification and claims of the present disclosure, unless the context clearly indicates otherwise, the words "a", "an", "the", and / or "this" are not limited in scope to the singular immediately preceding a word, but include both the singular and the plural. Generally, the terms "comprise", "comprising", "include", "including", "contain", "containing", "have", "having", and / or "hold", are intended to be inclusive, but not limiting, to the components, elements, steps, operations, and / or aspects that are explicitly identified. Such terms are not intended to be exclusive, but rather to allow for other features, elements, steps, operations, and / or aspects to be included.

[0049] These and other features and characteristics, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the disclosure. As such, it should be readily appreciated that the drawings are not necessarily drawn to scale in accuracy.

[0050] Various structural diagrams are used in the present disclosure to illustrate various variations according to embodiments of the present disclosure. It should be understood that the foregoing or the following structures are not intended to limit the present disclosure. The scope of protection of the present disclosure is subject to the claims.

[0051] Figure 1 is a schematic diagram of an intelligent device to which the present disclosure is applicable. Figure 1 The device shown includes a loading module in data connection with a GPU and a CPU, the loading module being used to load a sparse hybrid expert large language model from a device memory by the CPU to a GPU memory to perform an inference task, the selected experts and corresponding active neurons predicted by the sparse hybrid expert large language model.

[0052] The sparse hybrid expert large language model can activate only part of the model parameters to participate in calculation according to a given input. The sparse hybrid expert large language model generally replaces the FFN layer of a conventional dense large language model with a hybrid expert layer, each hybrid expert layer containing a certain number of experts and a gating unit to determine which experts to activate for calculation. The CPU (Central Processing Unit) is used for general-purpose computing and can directly manage the device memory (DRAM), which generally has a large capacity. The GPU (Graphics Processing Unit) contains a large number of computing cores and is good at handling parallel computing tasks, and neural network computation is generally handled by the GPU. The GPU has a built-in GPU memory with a small capacity. The loading module is a hardware or software module embedded in the intelligent device.

[0053] The sparse hybrid expert large language model generally has more model parameters than its dense counterpart, far exceeding the memory capacity of a consumer-grade GPU. For example, the currently optimal sparse hybrid expert large language model Mixtral 8x7B in terms of performance under the same model parameter quantity has 8 experts per hybrid expert layer and requires 94 GB of memory in half-precision representation. The NVIDIA RTX 3090Ti has only 24 GB of memory.

[0054] Offload techniques store experts in CPU DRAM to meet the memory limit of consumer-grade GPUs. However, this leads to unacceptable inference latency. As an example of Mixtral 8x7B: Mixtral-Offload stores experts in CPU DRAM and loads corresponding experts to GPU memory when needed. Although this makes it possible for the Mixtral 8x7B model to infer in a GPU with 24 GB of memory, the expert loading operation significantly increases inference latency, with an inference latency of about 3.9 seconds per word, significantly slower than the human reading speed of 0.2 to 0.3 seconds. Fiddler stores the experts of the Mixtral 8x7B model in CPU DRAM, and all computations related to experts are performed on the CPU side during inference. Although this avoids a large number of loading operations between the CPU and the GPU, since the computing power of the CPU is significantly lower than that of the GPU, the computation related to the experts also significantly increases the inference latency.

[0055] To solve the problem of significant inference latency of existing offloading schemes for sparse hybrid expert large language model inference tasks on consumer-grade GPUs, the present disclosure proposes an offloading scheme for sparse hybrid expert large language model inference tasks based on double-layer offloading. As shown in Figure 2 Compared with existing schemes that load all neurons densely into GPU memory, the present disclosure performs sparse loading based on the sparsity of neurons in two dimensions of coarse granularity and fine granularity, coarse granularity refers to the expert dimension, and fine granularity refers to the neuron dimension, where a neuron refers to a row or a column of a weight matrix. Specifically, during inference, the required experts and neurons are dynamically predicted, and the parallelism is improved by pre-fetching weights. Only part of the neurons in the experts are loaded for computation, thereby reducing the weight loading amount and improving the throughput. And maintain cache on GPU memory from both expert and neuron dimensions to minimize weight reloading amount, further improve inference efficiency.

[0056] Figure 3 is a flowchart of a sparse hybrid expert large language model inference task offloading method according to some embodiments of the present disclosure. In some embodiments, the sparse hybrid expert large language model inference task offloading method is performed by Figure 1 As shown in the offloading module, the sparse hybrid expert large language model inference task offloading method includes the following steps:

[0057] S310, predicting the experts selected by the sparse hybrid expert large language model in the inference task and the corresponding active neurons.

[0058] Prefetching is a method to reduce the latency of future usage of data by moving the data from slow storage to fast storage ahead of time. In large language model inference, it refers to moving the experts or neurons that will be needed in the future ahead of time into the GPU, and this moving process is executed in parallel with the computation process in the GPU. The prefetching method masks part of the computation latency, thereby reducing the inference latency.

[0059] However, in sparse hybrid expert large language models, both the selection of experts and the sparsity of neurons are dynamic. Only when the corresponding hybrid expert layer is executed can we get the exact selection of experts and active neurons. This sequence of execution makes it impossible for us to prefetch weights. In order to know the sparsity of neurons within an expert in advance, we need to make corresponding predictions. Previous work adds and trains a predictor for each layer, but if applied to sparse hybrid expert large language models, it will result in unacceptable additional memory overhead, especially in our scenario of memory-constrained consumer-grade GPUs.

[0060] In order to save memory, the present disclosure uses the weights of the model itself to predict the selection of experts and the sparsity of neurons. We add a predictor after the i-th attention layer for the prediction of the i+1-th hybrid expert layer, and the predictor structure is as shown in Figure 4 In the i-th predictor, the input needs to pass through the gate unit of the i+1-th layer to get the prediction of the i+1-th expert. Then the weight layer of the predicted expert is prefetched onto the GPU, and the input is calculated with the weight layer, and then the calculation result is passed through the activation function. The present disclosure selects the corresponding neurons with absolute values exceeding the threshold through the results of the activation function to get the prediction of active neurons.

[0061] By predicting the sparsity in advance, we can prefetch the required weights, mask part of the computation latency, and reduce the inference latency. During the entire prefetching process, the model inference is normally executed, and the present disclosure uses CUDA streams to realize the parallelism of weight loading and calculation. The prefetching of the i+1-th layer starts to execute at the same time as the hybrid expert layer of the i-th layer, and after the predictor of the i+2-th layer is executed, a global synchronization is performed until the i+1-th layer prefetching process is executed.

[0062] S320, the computing device loads the active neurons of the selected experts to execute the inference task.

[0063] After determining the experts and neurons needed for each inference, how to efficiently use GPU memory to reduce the loading of experts and neurons is a further consideration. Expert caching can effectively reduce the latency of using offloading methods to infer sparse hybrid expert large language models. Sparse hybrid expert large language models often reuse experts during multiple iterations of the inference process, so caching these reused experts in a location closer to the GPU's memory, such as the GPU HBM, can avoid the expensive retrieval from the CPU DRAM. Previous work has shown that the simplest LRU maintenance strategy for expert caching can still achieve satisfactory hit rates. In some embodiments of the present disclosure, a fixed amount of storage space is allocated in the GPU memory before the inference begins, which is used to cache a fixed number of experts, and is exempt from exceeding the memory limit of the GPU. During inference, the LRU strategy is used to maintain the cached experts. If an expert is selected and already stored in the cache, data loading can be avoided, thereby improving efficiency.

[0064] However, since the double-layer sparse loading we introduced when loading weights is not a complete loading of experts, even if the expert cache stores the expert selected by the current token, we cannot guarantee that the inference of this token is correct, because different inputs will produce different neuron sparsity. It is very likely that the neurons active during this token inference were not active during the last inference and were not loaded into the GPU expert cache. Therefore, we implement a coarse-to-fine caching strategy. As shown in Figure 5 For experts already stored in the expert cache, we only load the neurons that have an impact on the result and are not in the neuron cache, and update the expert's neuron cache by taking the union of the cached neurons and the active neurons during the current inference. For experts that are new to the expert cache, we directly perform neuron sparse loading and set the expert's neuron cache to the currently loaded neurons. In summary, the coarse-to-fine caching does hierarchical caching on both the expert and neuron dimensions, significantly reducing the weight loading latency caused by offloading.

[0065] Finally, the active neurons in the expert are compressed into a matrix on the CPU side, then the compressed matrix is loaded to the GPU side, and finally the decompression is performed on the GPU side. Double-layer sparse loading significantly reduces the weight loading amount and reduces the loading latency.

[0066] One embodiment of the present disclosure performs end-to-end experiments based on the edge device inference scenario based on the method described in S310-S320.

[0067] In the experiments, the input length is 64 and 128, while the output length varies from 8 to 512. The text generation sampling is performed on C4 and WikiText, which are high-quality text datasets covering a wide range of fields. The baseline implementation uses expert offloading. Specifically, the model weights are quantized to INT4, and the intermediate activations are stored as FP16. The non-mixed-expert layer weights and activations of the sparse-mixed-expert large language model are stored on the GPU, while the mixed-expert layer weights are stored in the DRAM of the CPU. This is a common sparse-mixed-expert large language model offloading strategy used in previous work. Figure 6 The comparison of end-to-end generation acceleration under different input and output lengths is shown. For Mixtral 8x7B, the average acceleration is 1.92x and 1.99x for input lengths of 64 and 128, respectively, while for DeepSeekMoE 16B, the average acceleration is 2.04x and 2.09x for the same input lengths, respectively. As the generation length increases, the acceleration effect of the method used in the present disclosure improves. This enhancement occurs because the number of generated tokens increases, the cache hit rate of the coarse-to-fine-grained cache system increases, and the weight loading delay is reduced.

[0068] The method used in the present disclosure predicts the usage of experts and the sparsity of neurons, and ignores some neurons during inference, which has a certain impact on the accuracy of the sparse-mixed-expert large language model. Figure 7 The accuracy with and without our method is compared. Experiments show that our method maintains competitive accuracy in different models and various downstream tasks. The accuracy of the sparse-mixed-expert large language model may fluctuate due to expert selection errors and the omission of some actual active neurons. However, sometimes this even improves the accuracy.

[0069] Figure 8 is a schematic diagram of a sparse-mixed-expert large language model inference task offloading device according to some embodiments of the present disclosure. As shown in Figure 8 the sparse-mixed-expert large language model inference task offloading device 800 includes a prediction module 810, an execution module 820. In some embodiments of the present disclosure, the sparse-mixed-expert large language model inference task offloading function is performed by Figure 1 the offloading module shown. Wherein:

[0070] The prediction module 810 is configured to predict the experts of the sparse-mixed-expert large language model selected in the inference task and the corresponding active neurons;

[0071] The execution module 820 is configured to load the active neurons of the selected experts on the computing device to perform the inference task.

[0072] In summary, the sparse mixed expert large language model inference task offloading method and device and equipment provided by the embodiments of the present disclosure reduce the inference delay of the sparse mixed expert large language model when offloading on a consumer-grade GPU when performing an inference task. In particular, by utilizing the double sparsity of the expert and neuron layers to optimize the weight loading process, the problem that existing methods cannot balance between inference delay and accuracy is solved. Through the coarse-to-fine prediction and prefetching mechanism, the required weights are accurately predicted without increasing the additional memory requirement, thereby realizing the overlap of weight loading and calculation to hide the delay, and solving the execution sequence problem in the sparse mixed expert large language model. Through the coarse-to-fine cache strategy, the demand for repeatedly loading weights is maximally reduced, and the problem of low efficiency of GPU memory utilization when offloading inference is solved.

[0073] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device and the module described above can refer to the corresponding description in the foregoing device embodiments, and will not be repeated here.

[0074] Although the subject matter described herein is provided in the general context of computer-executable instructions of a program module being executed by a computer system on a computing device, those skilled in the art will recognize that other implementations can be performed in combination with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Those skilled in the art will appreciate that the subject matter described herein can be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The subject matter can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0075] Those of ordinary skill in the art can realize that the units and method steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0076] It should be understood that the foregoing detailed description of the disclosure, rather than limiting the disclosure, is intended to exemplify and elicit the principles of the disclosure. Accordingly, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the disclosure should be included in the scope of protection of the disclosure. In addition, the claims of the disclosure are intended to cover all changes and modifications falling within the scope and boundary of the appended claims, or the equivalent form of such scope and boundary.

Claims

1. A method for offloading sparse mixed expert large language model inference tasks, characterized in that, The method comprises: predicting experts and corresponding active neurons of the sparse mixed expert large language model selected in an inference task; loading the active neurons of the selected experts in a computing device to perform the inference task; wherein the computing device comprises a CPU end and a GPU end, and the sparse mixed expert large language model comprises a plurality of expert layers, each expert layer comprising a plurality of experts, and the method further comprises: synchronously pre-fetching the active neurons of the selected experts of an (i+1)th expert layer on the GPU end when an ith expert layer performs the inference task, wherein i is a positive integer; the pre-fetching the active neurons of the selected experts of the (i+1)th expert layer on the GPU end comprises: synchronously predicting the selected experts of the (i+1)th expert layer based on the input of the ith expert layer; synchronously predicting the active neurons of the experts; synchronously pre-fetching the active neurons to the GPU end; the predicting the selected experts of the (i+1)th expert layer based on the input of the ith expert layer comprises: taking the hidden state output by an attention layer of the sparse mixed expert large language model as an input of a gating unit of the (i+1)th expert layer, and outputting the selected experts predicted by the (i+1)th expert layer, wherein the sparse mixed expert large language model comprises a layered structure, each layered structure is composed of an attention layer and an expert layer connected in series, the hidden state output by the attention layer is taken as an input of the expert layer, and the expert layer comprises a gating unit for activating the experts participating in calculation; the predicting the active neurons of the experts comprises: predicting the active neurons of the experts based on the weights of the experts; the predicting the active neurons of the experts based on the weights of the experts comprises: obtaining a weight matrix of the experts, activating the calculation result of the matrix multiplication of the weight matrix and the hidden state of the input by using an activation function, and taking the neurons corresponding to the activation results whose absolute values exceed a preset threshold as the active neurons.

2. The method of claim 1, wherein, The GPU end comprises an expert cache for caching experts and a neuron cache for caching the active neurons of each expert, and the loading the active neurons of the selected experts in the computing device comprises: if the selected experts are in the expert cache and the active neurons are in the neuron cache corresponding to the expert cache, reading the active neurons from the neuron cache to perform the inference task; if the selected experts are in the expert cache and the active neurons are not in the neuron cache corresponding to the expert cache, caching the active neurons in the neuron cache, and reading the active neurons from the neuron cache to perform the inference task; if the selected expert is not in the expert cache, caching the selected expert in the expert cache and caching the expert active neurons in the expert cache corresponding neuron cache, reading the active neurons from the neuron cache to perform the inference task.

3. The method of claim 2, wherein, the caching the selected expert in the expert cache includes: loading the selected expert into the expert cache based on LRU policy; the caching the expert active neurons in the expert cache corresponding neuron cache includes: when the selected expert is not in the expert cache, loading all the active neurons into the expert cache corresponding neuron cache; when the selected expert is in the expert cache, loading all the active neurons not in the expert cache corresponding neuron cache into the neuron cache.

4. The method of claim 3, wherein, the loading the active neurons includes: compressing the active neurons into a compressed matrix at the CPU end; loading the compressed matrix from the CPU end to the GPU end and decompressing.

5. An offloading device for sparse mixed expert large language model inference tasks, characterized in that, comprises: a prediction module configured to predict the selected expert and the corresponding active neurons of the sparse hybrid expert large language model in an inference task; an execution module configured to load the active neurons of the selected expert at a computing device to perform the inference task; wherein the computing device comprises a CPU end and a GPU end, the sparse hybrid expert large language model comprises a plurality of expert layers, each expert layer comprises a plurality of experts, and the apparatus further comprises: synchronously pre-fetching the active neurons of the selected expert of an (i+1)th expert layer at the GPU end when performing the inference task at an ith expert layer, wherein i is a positive integer; the synchronously pre-fetching the active neurons of the selected expert of the (i+1)th expert layer at the GPU end includes: synchronously predicting the selected expert of the (i+1)th expert layer based on the input of the ith expert layer; synchronously predicting the active neurons of the expert; synchronously pre-fetching the active neurons to the GPU end; the synchronously predicting the selected expert of the (i+1)th expert layer based on the input of the ith expert layer includes: outputting the hidden state of the ith attention layer of the sparse hybrid expert large language model as the input of the gating unit of the (i+1)th expert layer, and outputting the selected expert predicted by the (i+1)th expert layer, wherein the sparse hybrid expert large language model comprises a layered structure, each layered structure is composed of an attention layer and an expert layer connected in series, the hidden state output by the attention layer is used as the input of the expert layer, and the expert layer comprises a gating unit for activating the expert participating in calculation; the synchronously predicting the active neurons of the expert includes: predicting the active neurons of the expert based on the weight of the expert; the predicting the active neurons of the expert based on the weight of the expert includes: obtaining the weight matrix of the expert, The weight matrix and the matrix multiplication calculation result of the hidden state of the input are activated by using an activation function, and a neuron corresponding to an activation result exceeding a preset threshold in absolute value is taken as the active neuron.

Images