Model unloading method for sparse hybrid expert model and electronic device

Abstract

The invention provides a model unloading method of a sparse hybrid expert model and an electronic device. Based on the method and the device, the active expert module of each layer of the sparse hybrid expert model in the task execution process can be cached in the GPU, the activated expert module is called from the GPU cache when the task is executed, and when the activated expert module does not exist in the GPU cache, the module is loaded to the GPU cache. The loading and unloading processes of the expert module which is not activated in the model reasoning process are omitted, and the communication overhead and the storage overhead can be saved.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular to a model unloading method and electronic device for a sparse hybrid expert model. Background Art

[0002] With the rapid iteration and update of large language models, sparse mixture of experts (MoE) models have attracted much attention as a new model architecture. However, the number of parameters of sparse mixture of experts models increases exponentially with the increase of expert modules. Under the condition of the same number of layers, it will be several times or even dozens of times larger than the parameters of dense models, which may cause out of memory (OOM) problems. Model offloading technology can divide large language models into multiple sub-models or layers. Each time it is unloaded, only one layer or sub-model of the original model is loaded onto the device graphics card for inference, and then unloaded back to the host after completion, thereby reducing the demand for device graphics memory during deployment and inference of large models. However, when model offloading technology is used for sparse mixture of experts models, each load-inference-unload process will load and unload most of the expert modules that will not be activated, which increases communication overhead and storage overhead. Summary of the invention

[0003] Embodiments of the present application provide a model unloading method and electronic device for a sparse hybrid expert model.

[0004] In a first aspect, an embodiment of the present application provides a model unloading method for a sparse hybrid expert model, the method comprising:

[0005] When each layer of the sparse hybrid expert model is to perform a task, it is determined whether the first expert module cache of the layer includes the activated expert module of the layer, the first expert module cache is located in the GPU memory, the first expert module cache is used to store the active expert modules of the layer during the task execution process, and the activated expert modules are used to execute the task;

[0006] If the first expert module cache does not include any activated expert module, then the activated expert module that is not included is loaded from the host into the first expert module cache and then the task is executed based on the first expert module cache;

[0007] Alternatively, if the first expert module cache includes all activated expert modules, the task is executed.

[0008] Optionally, the expert modules in the first expert module cache are determined based on a top-K method, which selects top K expert modules with high activation probabilities as the expert modules in the first expert module cache based on the activation probabilities of the expert modules in this layer.

[0009] Optionally, before loading the activated expert module that is not included into the first expert module cache, the method further includes:

[0010] For each activated expert module that is not included, determining whether the first expert module cache is full;

[0011] If the first expert module cache is full, an expert module in the first expert module cache is eliminated.

[0012] Optionally, the expert modules in the first expert module cache are eliminated based on a first elimination strategy, and the first elimination strategy preferentially eliminates the expert module with the lowest response ratio in the first expert module cache. The response ratio of the expert model is determined based on the following formula:

[0013]

[0014] Among them, used_num is the number of times the expert module is activated and used, offload_num is the number of times the expert module is unloaded back to the host, and ratio represents the response ratio.

[0015] Optionally, loading the activated expert module that is not included into the first expert module cache includes:

[0016] Determining whether the second expert module cache includes the activated expert module that is not included;

[0017] If the second expert module cache includes the activated expert module that is not included, loading the activated expert module that is not included from the second expert module cache into the first expert module cache;

[0018] If the second expert module cache does not include the activated expert module that is not included, loading the activated expert module that is not included from the host into the first expert module cache;

[0019] The second expert module cache is located in the GPU memory, and is used to store the predicted expert modules that may be activated when the layer performs the task.

[0020] Optionally, the expert modules that may be activated are determined based on a first pre-fetching strategy, and the first pre-fetching strategy predicts the expert modules that may be activated in the layer based on the input features of the previous layer of the layer and the gating network of the layer.

[0021] Optionally, the method further includes:

[0022] Based on the first pre-fetch strategy, the possibly activated expert modules of the next layer of the layer are pre-fetched and put into the second expert module cache.

[0023] In a second aspect, an embodiment of the present application provides an electronic device, including:

[0024] A first judgment module is used to judge whether the first expert module cache of each layer of the sparse hybrid expert model includes the activated expert module of the layer when each layer is to perform a task, wherein the first expert module cache is located in the GPU memory, and the first expert module cache is used to store the active expert modules of the layer during the task execution process, and the activated expert module is used to execute the task;

[0025] a first loading module, configured to, when the first expert module cache does not include any of the activated expert modules, load the activated expert modules that are not included into the first expert module cache and then execute the task based on the first expert module cache;

[0026] An execution module is used to execute the task when the first expert module cache includes all the activated expert modules.

[0027] Optionally, the expert modules in the first expert module cache are determined based on a top-K method, which selects top K expert modules with high activation probabilities as the expert modules in the first expert module cache based on the activation probabilities of the expert modules in this layer.

[0028] Optionally, the electronic device further includes:

[0029] a first judgment module, used for judging whether the first expert module cache is full for each non-included activated expert module before loading the non-included activated expert module into the first expert module cache;

[0030] The elimination module is used to eliminate an expert module in the first expert module cache when the first expert module cache is full.

[0031] Optionally, the expert modules in the first expert module cache are eliminated based on a first elimination strategy, and the first elimination strategy preferentially eliminates the expert module with the lowest response ratio in the first expert module cache. The response ratio of the expert model is determined based on the following formula:

[0032]

[0033] Among them, used_num is the number of times the expert module is activated and used, offload_num is the number of times the expert module is unloaded back to the host, and ratio represents the response ratio.

[0034] Optionally, the electronic device further includes:

[0035] a second judgment module, used for judging whether the second expert module cache includes the activated expert module that is not included before loading the activated expert module that is not included into the first expert module cache;

[0036] The first loading module is further used for, when the second expert module cache includes the activated expert module that is not included, loading the activated expert module that is not included from the second expert module cache into the first expert module cache; or, when the second expert module cache does not include the activated expert module that is not included, loading the activated expert module that is not included from the host into the first expert module cache;

[0037] The second expert module cache is located in the GPU memory, and is used to store the predicted expert modules that may be activated when the layer performs the task.

[0038] Optionally, the expert modules that may be activated are determined based on a first pre-fetching strategy, and the first pre-fetching strategy predicts the expert modules that may be activated in the layer based on the input features of the previous layer of the layer and the gating network of the layer.

[0039] Optionally, the electronic device further includes:

[0040] The pre-fetch module is used to pre-fetch the possibly activated expert modules of the next layer of the layer based on the first pre-fetch strategy and put them into the second expert module cache.

[0041] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, at least one processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a method as described in any one of the above-mentioned first aspects is implemented.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described in any one of the above-mentioned first aspects is implemented.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes any one of the methods described in the first aspect.

[0044] Based on the present application, the active expert modules of each layer of the sparse hybrid expert model during the task execution can be cached in the GPU, and the activated expert modules can be called from the GPU cache when the task is executed. When the activated expert module does not exist in the GPU cache, the module is loaded into the GPU cache, thereby eliminating the loading and unloading process of the inactivated expert modules during the model reasoning process, which helps to save communication overhead and storage overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments.

[0046] Figure 1 It is a flow chart of the model unloading method of the sparse hybrid expert model provided in the embodiment of the present application.

[0047] Figure 2 This is an example diagram of a model unloading method for a sparse hybrid expert model provided in an embodiment of the present application.

[0048] Figure 3 This is an example diagram of a model unloading method for a sparse hybrid expert model provided in another embodiment of the present application.

[0049] Figure 4 This is an example diagram of a model unloading method for a sparse hybrid expert model provided in another embodiment of the present application.

[0050] Figure 5 It is a graph of reasoning index results of different elimination algorithms provided in the embodiments of the present application.

[0051] Figure 6A-6B It is a heat map of the expert module in the model unloading method of the sparse hybrid expert model provided in the embodiment of the present application.

[0052] Figure 7A-7B It is a schematic diagram of the results of an ablation experiment of the model unloading method of the sparse mixed expert model provided in an embodiment of the present application.

[0053] Figure 8 This is a graph of experimental results on the effect of the output length of the model unloading method of the sparse mixed expert model provided in an embodiment of the present application on the inference performance.

[0054] Fig. 9 This is a graph of experimental results showing the effect of the number of unloaded experts on the expert hit rate in the model unloading method of the sparse hybrid expert model provided in an embodiment of the present application.

[0055] Fig.10 This is a graph of experimental results showing the effect of the number of unloading experts on the model inference speed in the model unloading method of the sparse hybrid expert model provided in an embodiment of the present application.

[0056] Fig.11 This is a graph of experimental results showing the effect of the number of unloading experts on the required GPU memory in the model unloading method for the sparse hybrid expert model provided in an embodiment of the present application.

[0057] Fig.12 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application.

[0058] Fig.13 It is a structural schematic diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0059] The technical solution of the embodiment of the present invention will be described below in conjunction with the accompanying drawings in the embodiment of the present invention.

[0060] With the rapid iteration and update of large language models, sparse hybrid expert models have attracted much attention as a new model architecture. Compared with the previous dense model architecture, the sparse hybrid expert model architecture adds a gate network and multiple expert modules (multilayer perceptron (MLP) layer) in each layer, and allows the gate network to decide which expert module layers will be activated based on the characteristics of the input data, where each expert module processes a specific subset of tasks. Therefore, compared with dense models, sparse hybrid expert models have higher computational efficiency, more flexible task processing capabilities, and stronger model expansion capabilities.

[0061] Hybrid expert models are currently widely used on edge devices, such as smart voice assistants, autonomous driving, and real-time monitoring systems. In these application scenarios, it is usually necessary to provide faster data processing capabilities and faster response speeds on resource-constrained edge devices. For example, smart voice assistants require faster natural language processing response speeds to provide users with a good experience. Autonomous driving systems require real-time processing of large amounts of sensor data to provide faster response speeds and stability.

[0062] However, the number of parameters of the sparse hybrid expert model increases exponentially with the increase of expert modules. Under the condition of the same number of layers, it will be several times or even more than ten times larger than the dense model parameters. This places higher requirements on the device's video memory. Because in the process of model reasoning, not only the parameter weights of the model need to be stored, but also the key-value (KV) values ​​in the reasoning process. For example, for an 8B model parameter, when using 16-bit precision, at least 16GB of graphics card memory is required to save the model parameters, and additional video memory must be allocated to the KV cache. Therefore, the memory overflow problem caused by insufficient video memory has always been a major problem in model deployment and execution of reasoning tasks.

[0063] There are two main methods to alleviate the memory overflow problem during model inference, namely model compression and model unloading. The distillation and pruning techniques in the model compression method can effectively convert the original model into a new model with fewer parameters, while the quantization technology can effectively reduce the number of bits for parameter storage without changing the number of model parameters. In general, the three technologies of model compression, distillation, pruning and quantization, can effectively reduce the demand for video memory required to save model parameters. But at the same time, the model compression method will weaken the downstream reasoning tasks of the model and reduce the model performance. Therefore, the compressed large model needs to be retrained to alleviate the problem of model performance degradation. However, retraining of large models usually consumes more equipment and resources, which increases unnecessary costs.

[0064] Unlike model compression methods, model offloading does not change the model architecture in principle and logic. Model offloading uses the autoregressive characteristics of large language models during linear inference to split the large language model into multiple sub-models or layers. Each time it is unloaded, only one layer or sub-model of the original model is loaded onto the device graphics card for inference, and then unloaded back to the host after completion. By repeatedly loading, inference, and unloading in sequence, a complete inference task is logically completed on the original model. Model offloading technology can effectively utilize the linear stacking characteristics of large models, reduce the demand for device graphics memory when deploying and inferring large models, and make it possible to deploy and infer large models with huge parameter amounts on resource-constrained devices. Moreover, the model offloading method does not change the model architecture, that is, it does not change the size of model parameters and the number of parameter bits, so it does not reduce the quality of downstream reasoning tasks of the model. Whether it is for dense models or sparse hybrid expert models, model offloading technology is a very effective reasoning framework technology.

[0065] However, it is important to note that, like the dense model, the sparse hybrid expert model will load a layer or sub-model onto the graphics card each time it is loaded, but only a small portion of the expert modules in the layer loaded onto the graphics card will be activated. From another perspective, each load-inference-unload process will load and unload most of the expert modules that will not be activated. Therefore, compared with the dense model, the communication overhead of the sparse hybrid expert model using model unloading technology is usually much larger, and the inactivated expert modules increase unnecessary storage overhead.

[0066] Based on this, Figure 1 As shown, an embodiment of the present application provides a model unloading method for a sparse hybrid expert model, which includes steps S110-S120.

[0067] In step S110, when each layer of the sparse hybrid expert model is to perform a task, it is determined whether the first expert module cache of the layer includes the expert module activated in the layer. The first expert module cache is located in the GPU memory. The first expert module cache is used to store the active expert modules of the layer during the task execution process. The activated expert modules are used to execute the task. As an example, the active expert module can also be called a hot expert module, so the first expert module cache can also be called a hot expert module cache.

[0068] In principle, the sparse hybrid expert model will be assigned to different subsets of expert modules according to different subtasks, which means that the model will not use the same subset of experts in a long token, that is, the subset of experts activated in the same layer depends on the specific task type. On some tasks, some experts may always be activated, while some experts may not be activated. Therefore, based on the characteristic that the expert modules in the same layer are partially activated, the active expert modules can be saved in the GPU memory (the first expert module cache) as a "cache" for the expert modules of future tokens (these expert modules temporarily cached in the GPU are likely to be activated again).

[0069] In step S120, if the first expert module cache does not include any activated expert module, the activated expert modules that are not included are loaded into the first expert module cache until the first expert module cache includes all activated expert modules and then the task is executed based on the first expert module cache; or, if the first expert module cache includes all activated expert modules, the task is executed.

[0070] When a hybrid sparse model is executed at a certain time, the required expert modules will be activated. If the same experts are stored in the first expert module cache, they can be directly called in the first expert module cache to calculate the reasoning task. If the activated expert module does not exist in the first expert module cache, it can be loaded (for example, from the host) into the first expert module cache. When all activated expert modules are in the first expert module cache, the reasoning task can be executed.

[0071] Based on the present application, the active expert modules of each layer of the sparse hybrid expert model during the task execution can be cached in the GPU, and the activated expert modules can be called from the GPU cache when the task is executed. When the activated expert module does not exist in the GPU cache, the module is loaded into the GPU cache, thereby eliminating the loading and unloading process of the inactivated expert modules during the model reasoning process, which helps to save communication overhead and storage overhead.

[0072] Optionally, the expert modules in the first expert module cache can be determined based on the top-K method. The top-K method selects the top K expert modules with high activation probabilities as the expert modules in the first expert module cache based on the activation probabilities of the expert modules at this layer. That is, K expert modules that are very likely to be activated are saved on the GPU. For example, the expert modules that are very likely to be activated can be determined according to the tasks executed at this layer. It should be noted that K << M, where M is the total number of expert modules at each layer of the sparse mixture of experts model. For example, K can be determined according to the number of activated expert modules when the sparse mixture of experts model executes tasks at each layer. Usually, when the sparse mixture of experts model executes tasks at each layer, the number of activated expert modules is fixed, and K can be greater than or equal to this fixed number. As an example, when the number of activated expert modules is 2 and the total number of expert modules at each layer is 8 when the sparse mixture of experts model executes tasks at each layer, then 2 <= K < 8. Another example is that K can also be determined according to the number of unloaded expert modules when the sparse mixture of experts model executes tasks at each layer. The number of unloaded expert modules here refers to the number of expert modules that are not loaded into the GPU when each layer of the model executes tasks. As an example, when the number of activated expert modules is 2, the number of unloaded experts is 5, and the total number of expert modules at each layer is 8 when the sparse mixture of experts model executes tasks at each layer, then K = 8 - 5 = 3.

[0073] When using the top-K method to determine the expert modules in the first expert module cache, how to perform the allocation and scheduling algorithm of the first expert module cache is a key point.

[0074] Based on this, optionally, before loading the activated expert modules that are not included into the first expert module cache, Figure 1 the method shown can also include: for each activated expert module that is not included, determining whether the first expert module cache is full; if the first expert module cache is full, then eliminating one expert module in the first expert module cache. That is, whenever an activated expert module that is not included is loaded into the first expert module cache, it is necessary to determine whether the first expert module cache is full. If it is full, then one expert module in the first expert module cache needs to be eliminated, and the eliminated module can be unloaded to the host. In this way, storage space can be freed up for the loaded expert modules, and more likely to be activated expert modules can also be retained in the first expert module cache.

[0075] Optionally, the expert modules in the first expert module cache can be eliminated based on the first elimination strategy. The first elimination strategy preferentially eliminates the expert module with the lowest response ratio in the first expert module cache. The response ratio of the expert model is determined based on the following formula:

[0076]

[0077] Among them, used_num is the number of times the expert module is activated and used, offload_num is the number of times the expert module is unloaded back to the host, and ratio represents the response ratio. The number of times the expert module is activated and used and the number of times the expert module is unloaded back to the host can be recorded during the process of the model performing the reasoning task. As an example, the first elimination strategy can be called a low response ratio first elimination algorithm (low response ratio next, LRRN).

[0078] Traditional elimination strategies, such as the least recently used (LRU) strategy and the least frequently used (LFU) strategy, play a vital role in computer cache management and scheduling. However, the LRU algorithm focuses on local usage in the recent period of time and fails to pay attention to global scheduling information. Although the LFU algorithm can obtain global scheduling information through usage frequency, it cannot scientifically measure the indicators for expert module elimination. The elimination strategy of expert modules that meet expectations should pay attention to both the usage frequency and the unloading frequency of expert modules. Therefore, it is necessary to keep expert modules with high usage frequency and low unloading frequency in the GPU memory as much as possible, because they are likely to be activated again in the next reasoning.

[0079] According to the definition of the response ratio shown in formula (1), the expert module with low activation frequency and high unloading frequency has a lower ratio, and will be preferentially unloaded from the GPU memory to the host, while the expert module with high activation frequency and low unloading frequency has a higher ratio, and will be preferentially retained in the GPU memory. This is in line with the expected expectations of the elimination strategy, because a low activation frequency indicates that the expert module is not often activated, and a high unloading frequency indicates that the expert module is often unloaded during the reasoning process. The combination of the two can well select the target expert module for elimination.

[0080] The model offloading technology needs to ensure that all activated expert modules of this task are in the GPU cache before the calculation of the reasoning task of this layer is performed. The operation of ensuring that all activated expert modules are in the GPU cache generally involves two communication operations, namely, the operation of unloading the eliminated expert modules from the GPU to the host and the operation of loading the missing activated expert modules from the host to the GPU. These two operations do not overlap with the reasoning task calculation of this layer and need to be executed in strict sequence. Although LRRN can reduce the average expert loading time, most of the reasoning time is still spent waiting to load the activated expert modules. Therefore, in order to solve this problem, we can refer to the characteristics of the conventional dense transformer architecture when using offloading technology. Based on the characteristics of linear autoregression, the dense model can preload the next layer in advance while the previous layer is still running, but this method is no longer applicable to the sparse hybrid expert model. In the sparse hybrid expert model, each layer needs to obtain the input features after the calculation of the previous layer is completed, and then the gating network determines which experts should be loaded in the current layer. This means that it is difficult to parallelize the reasoning calculation of the previous layer and the prefetch of the next layer.

[0081] Although it is not possible to reliably pre-fetch the expert modules that will be activated in the next layer, it is still possible to try to guess the expert modules that may be activated in the next layer and speculatively load them while processing the reasoning task of the previous layer. If the guess is correct, the reasoning of the next layer will be accelerated. If the guess is incorrect, the activated experts are loaded from the host to the GPU memory. In other words, this type of speculative loading will not change the final model reasoning task, but if the guess is accurate enough, it can still reduce latency.

[0082] Based on this, optionally, loading the non-included activated expert module into the first expert module cache may include: determining whether the second expert module cache includes the non-included activated expert module; if the second expert module cache includes the non-included activated expert module, loading the non-included activated expert module from the second expert module cache into the first expert module cache; if the second expert module cache does not include the non-included activated expert module, loading the non-included activated expert module from the host into the first expert module cache; wherein the second expert module cache is located in the GPU memory, and the second expert module cache is used to store the predicted expert modules that may be activated when the layer executes the task. As an example, the second expert module cache may also be referred to as a pre-fetch expert module cache.

[0083] That is, for each layer of the sparse hybrid expert model, the expert modules that are predicted to be activated when executing the task can also be cached in the GPU memory (second expert module cache). When the first expert module cache does not include the activated expert module, it can be determined whether the second expert module cache includes the expert module. If so, the expert module is loaded from the second expert module cache to the first expert module cache, which helps to accelerate the inference calculation of the model.

[0084] Furthermore, the expert modules that may be activated may be determined based on a first pre-fetching strategy, and the first pre-fetching strategy may predict the expert modules that may be activated in the layer based on the input features of the previous layer of the layer and the gating network of the layer.

[0085] When analyzing the sparse mixed expert model, it is not difficult to find that the Transformer layer uses the residual connection technology, that is, y = F (x) + x, and each layer adds the previous hidden layer features. In other words, the output features of each layer carry the output feature information of the previous layer. Using the residual characteristics, the gating network of the sparse mixed expert model determines the subset of activated expert modules of the layer based on the output features of the previous layer. It can be indirectly converted into: using the input features of the previous layer and the gating network of the next layer to predict the activated expert modules of the next layer, without letting the gating network of the next layer wait until the output of the previous layer to make a decision. This simple adjustment and conversion introduces an inductive bias, so that the hidden state of any layer can make an appropriate estimate of the hidden state of the next layer. Therefore, based on the input features of the previous layer of a certain layer and the gating network of this layer, it is possible to effectively predict the expert modules that may be activated in this layer.

[0086] Further, Figure 1 The method shown may also include: based on the first pre-fetch strategy, pre-fetching the expert modules of the next layer that may be activated and putting them into the second expert module cache. It is worth noting that pre-fetching the expert modules of the next layer can be performed simultaneously with executing the tasks of the layer. In this way, the pre-fetching operation and the task calculation operation can overlap, which helps to improve efficiency.

[0087] Combine the following Figure 2-Figure 4 , the model unloading method of the sparse hybrid expert model of the embodiment of the present application is introduced by example.

[0088] like Figure 2 , which is a flow chart of a model unloading method of a sparse hybrid expert model based on a first elimination strategy. It is worth noting that in the following description, the first elimination strategy is referred to as the LRRN elimination strategy.

[0089] exist Figure 2In the method shown, the top-K method is used to save K expert modules that are very likely to be activated on the GPU, where K << M, and M is the total number of expert modules on each layer of the sparse mixture of experts model. Suppose the sparse mixture of experts model has N layers, and the size of each expert module is S. Then, to cache the hot expert modules of the entire model, GPU memory of size N * K * S needs to be allocated.

[0090] If K > M, it means that the GPU memory can accommodate the entire sparse mixture of experts model, which degenerates into an ordinary offloading technique. Therefore, it is naturally necessary to design K << M. It should be noted that there must be a situation where the t experts (t < K) activated this time are not included in the K cached experts. Then, the expert modules not hit in the GPU cache need to be transferred from the main device to the GPU memory. Only when all the expert modules activated this time are in the GPU cache, the inference task calculation is performed. As an example, Figure 2 The pseudocode of the expert module scheduling algorithm in the method shown can be as shown in Table 1. It should be noted that in Figure 2 In the method shown, the status of each layer (such as the hit count and miss count) can be recorded for the study of the inference process, which does not limit the embodiments of the present application. The hit count is used to represent the number of times an activated expert module is hit in the first expert module cache of this layer, and the miss count is used to represent the number of times an activated expert module is not hit in the first expert module cache of this layer.

[0091] Table 1 Expert Module Scheduling Algorithm Based on LRRN

[0092]

[0093]

[0094] As Figure 3 shown, it is a schematic flowchart of the model offloading method of the sparse mixture of experts model based on the first elimination strategy and the first prefetching strategy. It should be noted that in the following description, the first elimination strategy is called the LRRN elimination strategy, and the first prefetching strategy is called the speculative prefetching technique. As an example, Figure 3 The pseudocode of the expert module scheduling algorithm in the method shown can be as shown in Table 2.

[0095] In Figure 3In the method shown, the top-K method is used to save K most likely to be activated expert modules on the GPU, where K << M, and M is the total number of expert modules on each layer of the sparse mixture of experts model. Suppose the sparse mixture of experts model has N layers, and the size of each expert module is S. Then, to cache the hot expert modules of the entire model, GPU memory with a size of N * K * S needs to be allocated. In addition, according to the speculative prefetching technique, the possibly activated expert modules of each layer are prefetched into the second expert module cache. It should be noted that in Figure 3 the method shown, the status of each layer (such as the hit count and miss count) can be recorded for the study of the inference process, which does not limit the embodiments of this application. The hit count is used to represent the number of times the activated expert module is hit in the first expert module cache of this layer, and the miss count is used to represent the number of times the activated expert module is not hit in the first expert module cache of this layer.

[0096] Table 2 Expert Scheduling Algorithm Based on LRRN and the First Prefetching Strategy

[0097]

[0098]

[0099] As Figure 4 shown, it is an example diagram of the model offloading method of the sparse mixture of experts model based on the first elimination strategy and the first prefetching strategy.

[0100] Figure 4 It includes a simple schematic diagram of the sparse mixture of experts model. When expanding the nth layer, it can be seen that each layer mainly includes an attention layer, a residual layer, a normalization layer, and a MoE expert layer. Its detailed layer architecture is as shown in the expanded sub-diagram.

[0101] According to the model unloading method of the sparse hybrid expert model of the embodiment of the present application, while completing the inference calculation of the n-1 layer layer, the expert group of the n layer layer will be pre-fetched according to the first pre-fetch strategy and placed in the prefetch_cache of the GPU, that is, the expert feed-forward neural network (FFN1) and FFN2. When preparing to start the inference calculation of the i-th layer, according to the probability distribution calculated by the gating network, it is calculated that FFN1 and FFN2 of the n-th layer layer are to be activated. Since the expert FFN1 has been retained in the first expert module cache of the GPU layer (n) in the previous inference calculation, it does not need to be loaded again. However, the activated expert FFN2 is not in the first expert module cache of the GPU layer (n), so a loading operation is required. Since the first expert module cache of the GPU layer (n) is full, it is necessary to perform expert unloading according to the low response ratio priority elimination strategy LRRN. Here, it is assumed that the response ratio of the expert FFNi is the lowest, so the expert FFNi will be unloaded back to the host first. Since FFN2 has been successfully prefetched and placed in prefetch_cache when performing inference calculation of layer n-1, expert FFN2 can be loaded from prefetc_cache. Only when both expert FFN1 and FFN2 are in the first expert module cache of layer(n) of GPU, inference calculation of layer n can be performed, and expert group of layer n+1 can be prefetched and placed in prefetch_cache.

[0102] Combine the following Figure 5-Figure 11 , an experimental study is conducted on the model unloading method of the sparse hybrid expert model of the embodiment of the present application.

[0103] In order to evaluate the effectiveness of the model unloading method of the sparse mixed expert model of the embodiment of the present application, the focus is on the currently open source and representative sparse mixed expert model Mixtral-8x7B-Instruct, which has 8 experts in each layer layer and a total parameter volume of 46.7B, of which the parameter volume of the expert module accounts for 96.6% of the total parameter volume, and the other parameters are allocated to the embedding layer, attention layer and layer normalization (LayerNorm), etc. In order to be able to implement reasoning on low-end graphics cards, the Mixtral-8x7B-Instruct-v0.1 open source model provided by the Mixtral team was used as the target model of the experiment. At the same time, common reasoning tasks such as writing, translation, (Python, Java) code generation, literature summary, role-playing, and daily conversations were set to test and verify the effectiveness of the model unloading method of the sparse mixed expert model of the embodiment of the present application. The host device used in the experiment is equipped with an NVIDIA RTX3090 graphics card with a video memory of 24G.

[0104] Model inference speed results analysis

[0105] The model unloading method of the sparse hybrid expert model in the embodiment of the present application adopts the low response ratio priority elimination strategy LRRN to eliminate and unload experts. In order to better highlight the advantages of the LRRN strategy, the experiment also implements two other classic traditional elimination strategies - LFU and LRU. And let LRRN compare with these two traditional elimination algorithms in terms of model inference speed indicators. The comparison results can be shown in Table 3 and Figure 5 , Figure 6A-6B shown.

[0106] Table 3 Average reasoning performance with fixed reasoning output length of 512 and 5 experts per layer

[0107]

[0108] As shown in Table 3 and Figure 5 As shown in the figure, the reasoning performance obtained by using the LFU strategy and the LRU strategy for expert elimination and unloading is similar. The average reasoning time per prompt of LFU is 260 seconds, and the reasoning speed is 2.39 tokens / s on average. The average reasoning time per prompt of LRU is 258 seconds, and the reasoning speed is 2.40 tokens / s on average. Compared with LFU and LRU, LRRN achieves better reasoning performance. The average reasoning time per prompt of LRRN is only 197 seconds, and the reasoning speed is as high as 3.21 tokens / s on average.

[0109] It is worth noting that in the above experiment, the number of experts unloaded from each layer of the model is controlled to 5. As mentioned above, the number of expert modules K cached in each layer in the first expert module cache can generally be set according to the number of expert modules activated by each layer of the model to perform tasks. Usually, the number of expert modules activated by each layer to perform tasks is fixed, so the number of expert modules cached in each layer in the first expert module cache can be controlled by controlling the number of experts unloaded from each layer of the model. For example, the number of expert modules in each layer of the model is 8, the number of expert modules activated by each layer of the model to perform tasks is 2, and the number of experts unloaded from each layer of the model is controlled to 5, then the size of K is 8-5=3, and K>2 at this time.

[0110] Fig. 6A The figure shows the heat map of the number of times the 8 experts in each layer are unloaded from the GPU to the host when the sparse mixed sparse model completes all the reasoning tasks in the above experiment. The darker the color, the higher the unloading frequency of the expert. It can be clearly seen that at least two experts in each layer have a low unloading frequency, which means that once these experts with low unloading frequency are loaded into the GPU memory during the reasoning process, they are very unlikely to be eliminated and unloaded back to the host.

[0111] Figure 6B The figure shows the heat map of the number of times the eight experts in each layer are activated when the sparse mixed sparse model completes all reasoning tasks in the above experiment. The darker the color, the higher the number of times the expert is activated. It can be clearly seen that several experts in each layer are frequently activated.

[0112] Analysis of ablation experiment results

[0113] In order to better study the model unloading method of the sparse hybrid expert model of the embodiment of the present application, an ablation experiment was conducted on the improvement strategy of each part. The experimental results can be shown in Table 4 and Figure 7A-7B shown.

[0114] Among them, local offloading technology means using traditional offloading technology to perform offloading scheduling between the host and GPU, and loading all expert modules of the layer onto the GPU only when needed; using GPU cache expert strategy means retaining a fixed number of expert caches for each layer in the GPU; prefetching technology means using the first prefetching strategy to prefetch the expert module combination that may be activated in the next layer in advance; LRRN means using the LRRN elimination algorithm to manage the expert module cache in the GPU.

[0115] As shown in Table 4 and Fig. 7AAs shown in the figure, adopting different strategies in turn improves the speed of model reasoning. When all optimization strategies are adopted, the model reasoning speed can reach 3.21 tokens / s, which is much better than the reasoning speed of 1.27 tokens / s of the local offloading technology without adopting any optimization strategy.

[0116] In order to intuitively see the effect of using various optimization strategies on improving inference performance, assuming that the speedup ratio of using local offloading technology is 1.0, then the speedup ratio corresponding to other optimization strategies should be the ratio of the inference speed corresponding to other optimization strategies to the inference speed of local offloading technology. Figure 7B It can be seen that the three optimization strategies of GPU cache expert, prefetch technology and LRRN elimination strategy have the highest acceleration ratio, which is as high as 2.53.

[0117] Table 4 Reasoning performance using different strategies

[0118]

[0119] Exploring the impact of output length on inference performance

[0120] In order to quantitatively analyze and observe the performance change trend of the low response ratio priority elimination strategy LRRN at different reasoning output lengths, the number of expert unloadings per layer is fixed to 5, and the reasoning output length is changed from 64 short text output to 1024 long text output for experiments. The experimental results can be shown as follows Figure 8 shown.

[0121] from Figure 8 It can be seen that the change of the inference output length does not affect the performance of the LRRN elimination algorithm. When the output length changes, the hit rate of each layer of the expert module remains relatively stable, always floating around the baseline of 40% hit rate.

[0122] Exploring the impact of the number of offloaded experts per layer on inference performance

[0123] In order to quantitatively analyze and observe the performance trend of the LRRN algorithm with different numbers of offloading experts per layer, the inference output length is fixed to 512, and the number of offloading experts per layer is reduced from a high offloading number of 6 to a low offloading number of 3. The experimental results can be shown as follows Figure 9-11 shown.

[0124] from Fig. 9 It can be seen that as the number of expert unloadings per layer decreases, the hit rate of the expert module cache per layer will increase. This is also in line with experimental expectations, because the reduction in the number of expert unloadings per layer means that the number of experts retained in the GPU cache will increase accordingly, so that the probability of experts being hit and activated in the GPU cache in the next reasoning task will increase accordingly.

[0125] It can be seen that when the number of experts unloaded per layer is set to 3, that is, each layer needs to unload 3 expert modules back to the host, the expert hit rate of each layer reaches more than 60%, and the highest reaches 74%. However, when the number of experts unloaded per layer is set to 6, the expert hit rate of each layer is reduced to about 30%. On the whole, considering the size of GPU memory and the hit rate of LRRN, the appropriate number of unloaded experts per layer can be set according to the resource situation of the device.

[0126] from Fig.10 It can be seen that as the number of experts unloaded per layer increases, the corresponding model reasoning speed also decreases. When the number of unloaded experts per layer is set to 3, the model reasoning speed can reach 4.60+tokens / s. When the number of unloaded experts per layer is set to 6, the reasoning speed drops to 2.80tokens / s. This is also in line with the expectations of the experiment, because Fig. 9 As analyzed, as the number of unloaded experts per layer increases, the expert hit rate of each layer decreases accordingly. Then, more communication overhead is required in the operation of calling the expert module from the host to the GPU, so its inference speed will also decrease accordingly.

[0127] from Fig.11 It can be seen that as the number of experts unloaded per layer increases, the memory required for reasoning becomes less and less. For example, when the number of expert modules unloaded per layer is set to 0, that is, no expert unloading is performed on the model, and all experts are retained in the GPU memory, then the GPU memory required for reasoning is 20.74GB; compared with the number of expert modules unloaded per layer being set to 5, that is, the number of expert modules unloaded per layer is 5, and only 3 experts are retained in the GPU memory per layer, then the GPU memory required for reasoning is only 10.33GB, which is much lower than the 20.74GB GPU memory required when no expert unloading is performed. This can effectively prove that the model unloading method of the sparse hybrid expert model of the embodiment of the present application can effectively reduce the GPU memory overhead during model reasoning.

[0128] The method provided in the embodiment of the present application is described in detail above. The device provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0129] See also Fig.12 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.12 The electronic device 1200 shown may include:

[0130] The first judgment module 1210 may be used to judge whether the first expert module cache of each layer of the sparse hybrid expert model includes the activated expert module of the layer when each layer is to perform a task, wherein the first expert module cache is located in the GPU memory, and the first expert module cache is used to store the active expert modules of the layer during the task execution process, and the activated expert modules are used to execute the task;

[0131] The first loading module 1220 may be configured to, when the first expert module cache does not include any of the activated expert modules, load the activated expert modules that are not included into the first expert module cache, until the first expert module cache includes all the activated expert modules and then executes the task based on the first expert module cache;

[0132] The execution module 1230 may be configured to execute the task when the first expert module cache includes all the activated expert modules.

[0133] Optionally, the expert modules in the first expert module cache may be determined based on a top-K method, which selects top K expert modules with high activation probabilities as the expert modules in the first expert module cache based on the activation probabilities of the expert modules in the layer.

[0134] Optionally, the electronic device 1200 may further include:

[0135] The first determination module 1240 may be configured to determine whether the first expert module cache is full for each non-included activated expert module before loading the non-included activated expert module into the first expert module cache;

[0136] The elimination module 1250 may be configured to eliminate an expert module in the first expert module cache when the first expert module cache is full.

[0137] Optionally, the expert modules in the first expert module cache may be eliminated based on a first elimination strategy, wherein the first elimination strategy preferentially eliminates the expert module with the lowest response ratio in the first expert module cache, and the response ratio of the expert model is determined based on the following formula:

[0138]

[0139] Among them, used_num is the number of times the expert module is activated and used, offload_num is the number of times the expert module is unloaded back to the host, and ratio represents the response ratio.

[0140] Optionally, the electronic device 1200 may further include:

[0141] The second judgment module 1260 may be used to judge whether the second expert module cache includes the activated expert module that is not included before loading the activated expert module that is not included into the first expert module cache;

[0142] The first loading module 1220 may also be used to load the non-included activated expert module from the second expert module cache into the first expert module cache when the non-included activated expert module is included in the second expert module cache; or, if the non-included activated expert module is not included in the second expert module cache, load the non-included activated expert module from the host into the first expert module cache;

[0143] The second expert module cache is located in the GPU memory, and is used to store the predicted expert modules that may be activated when the layer performs the task.

[0144] Optionally, the expert modules that may be activated may be determined based on a first pre-fetching strategy, which predicts the expert modules that may be activated in the layer based on the input features of the previous layer of the layer and the gating network of the layer.

[0145] Optionally, the electronic device 1200 may further include:

[0146] The pre-fetch module 1270 is used to pre-fetch the possibly activated expert modules of the next layer of the layer based on the first pre-fetch strategy and put them into the second expert module cache.

[0147] The present application also provides an electronic device. Fig.13 As shown, the electronic device 1300 includes: at least one processor 1301, a memory 1302, and a computer program 1303 stored in the memory 1302 and executable on the at least one processor 1301. When the processor 1301 executes the computer program 1303, the method provided in the present application is implemented.

[0148] Exemplarily, the computer program 1303 may be divided into one or more modules / units, one or more modules / units are stored in the memory 1302, and executed by the processor 1301 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 1300.

[0149] Those skilled in the art will understand that Fig.13It is only an example of the electronic device 1300 and does not constitute a limitation of the electronic device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 1300 may also include input and output devices, network access devices, buses, etc.

[0150] The processor 1301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0151] The memory 1302 may be an internal storage unit of the electronic device 1300, such as a hard disk or memory of the electronic device 1300. The memory 1302 may also be an external storage device of the electronic device 1300, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 1300. Further, the memory 1302 may also include both an internal storage unit of the electronic device 1300 and an external storage device. The memory 1302 is used to store computer programs and other programs and data required by the electronic device 1300. The memory 1302 may also be used to temporarily store data that has been output or is to be output.

[0152] The electronic device 1300 provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0153] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0154] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above method embodiments, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, ROM (read-only memory), RAM (random access memory), CD-ROM (compact disc read-only memory), magnetic tape, floppy disk and optical data storage device. The computer-readable storage medium mentioned in the present application can be a non-volatile storage medium, in other words, a non-transient storage medium.

[0156] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0157] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0158] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0159] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0160] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0161] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0162] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A model unloading method for a sparse mixed expert model, characterized in that: include: When each layer of the sparse hybrid expert model is to perform a task, it is determined whether the first expert module cache of the layer includes the expert module activated in the layer, the first expert module cache is located in the memory of the graphics processor GPU, the first expert module cache is used to store the active expert modules of the layer in the process of performing the task, and the activated expert module is used to perform the task; If the first expert module cache does not include any of the activated expert modules, the activated expert modules that are not included are loaded into the first expert module cache until the first expert module cache includes all the activated expert modules and then the task is executed based on the first expert module cache; Alternatively, if the first expert module cache includes all the activated expert modules, the task is executed.

2. The method according to claim 1, characterized in that The expert modules in the first expert module cache are determined based on a top-K method, which selects the top K expert modules with high activation probabilities as the expert modules in the first expert module cache based on the activation probabilities of the expert modules in this layer.

3. The method according to claim 2, characterized in that Before loading the activated expert module that is not included into the first expert module cache, the method further includes: For each of the activated expert modules that are not included, determining whether the first expert module cache is full; If the first expert module cache is full, an expert module in the first expert module cache is eliminated.

4. The method according to claim 3, characterized in that The expert modules in the first expert module cache are eliminated based on a first elimination strategy, wherein the first elimination strategy preferentially eliminates the expert module with the lowest response ratio in the first expert module cache, and the response ratio of the expert model is determined based on the following formula: Wherein, the used_num is the number of times the expert module is activated and used, the offload_num is the number of times the expert module is unloaded back to the host, and the ratio represents the response ratio.

5. The method according to claim 4, characterized in that The step of loading the activated expert module that is not included into the first expert module cache comprises: Determining whether the second expert module cache includes the activated expert module that is not included; If the second expert module cache includes the activated expert module that is not included, loading the activated expert module that is not included from the second expert module cache into the first expert module cache; If the second expert module cache does not include the activated expert module that is not included, loading the activated expert module that is not included from the host into the first expert module cache; The second expert module cache is located in the GPU memory, and is used to store the predicted expert modules that may be activated when the layer performs the task.

6. The method according to claim 5, characterized in that The expert module that may be activated is determined based on a first pre-fetching strategy, and the first pre-fetching strategy predicts the expert module that may be activated in the layer based on the input features of the previous layer of the layer and the gating network of the layer.

7. The method according to claim 6, characterized in that The method further comprises: Based on the first pre-fetch strategy, the possibly activated expert modules of the next layer of the layer are pre-fetched and put into the second expert module cache.

8. An electronic device, characterized in that: include: A first judgment module is used to judge whether the first expert module cache of each layer of the sparse hybrid expert model includes the activated expert module of the layer when each layer is to perform a task, wherein the first expert module cache is located in the memory of the graphics processor GPU, and the first expert module cache is used to store the active expert modules of the layer during the task execution process, and the activated expert module is used to execute the task; A first loading module, configured to, when the first expert module cache does not include any of the activated expert modules, load the activated expert modules that are not included into the first expert module cache; An execution module is used to execute the task when the first expert module cache includes all the activated expert modules.

9. An electronic device comprising a memory, at least one processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images