A terminal device-oriented sparse deployment method of a large language model

Abstract

The application relates to the fields of artificial intelligence and mobile computing technology, and discloses a terminal device-oriented sparse large language model deployment method, which is improved on the basis of a Switch Transformer network structure through probability statistics and parallelization technology. In the Switch Transformer network structure, an FFN layer in a T5 model is replaced by an MoE structure, a global expert temporary register is designed, the expert temporary register manages expert networks loaded from a disk, and when reasoning is performed, the expert networks in the expert temporary register are directly used for reasoning, so that a model larger than the upper limit of the memory of a terminal device is deployed on the terminal device at the cost of a small increase in reasoning time, and zero precision loss is achieved in the reasoning deployment.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and mobile computing technology, specifically to a method for deploying sparse large language models for terminal devices. Background Technology

[0002] Most mainstream large language models are based on the Transformer model. The Transformer model consists of an encoder and a decoder, with the self-attention structure and the FFN layer being the most important components. GPT and BERT, two of the most popular large language models, are derived from Transformers using only the decoder and encoder, respectively. There are also models with both encoders and decoders, such as T5. However, these large language models are non-sparse; all model weights are activated during inference. To scale the model further, researchers have proposed sparse large language models based on non-sparse models. The Switch Transformer is a sparse large language model developed from T5. The Switch Transformer uses a Mixture-of-Experts (MoE) mechanism to extend the FFN layer into a structure where multiple experts are selected through gating layers. This allows the number of parameters in the large oracle model to be increased by tens to hundreds of times, thereby reducing training costs and achieving better performance.

[0003] Current large language models are deployed on cloud servers. Users upload their input questions to the cloud via the network, and the cloud server performs inference and downloads the results to the user. The main reason large language models cannot be directly deployed to end devices is their massive parameter size, which is far too large compared to the memory requirements of end devices. Therefore, the parameters of large language models need to be compressed for deployment on end devices. Sparse large language models based on MoE, such as the Switch Transformer, only activate a subset of parameters during each inference iteration. Parameters not selected by the gating layer remain inactive and have no impact on the inference results. Therefore, these types of sparse large language models based on MoE can have their parameter size compressed by loading only the activated parameters, thus achieving deployment on end devices.

[0004] This invention targets sparse large language models based on MoE, enabling deployment on terminal devices such as personal computers and smart terminals. This invention does not alter the model parameters and guarantees zero precision loss. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a method for deploying sparse large language models for terminal devices. This method has the advantages of being able to deploy on terminal devices such as personal computers and smart terminals without changing the model parameters and ensuring zero precision loss, thus solving the problems mentioned in the background section.

[0007] (II) Technical Solution

[0008] This invention provides the following technical solution: a method for deploying sparse large language models for terminal devices, which improves upon the Switch Transformer network structure through probability statistics and parallelization techniques, so as to deploy a model far exceeding the memory limit of the terminal device at the cost of a small increase in inference time, and achieve inference deployment with zero precision loss.

[0009] In the Switch Transformer network structure, the FFN layer in the T5 model is replaced with the MoE structure;

[0010] Before the model performs inference on the cloud server, all parameters of the Switch Transformer model are stored in memory. During inference, all parameters of all layers except the MoE layer are activated.

[0011] Deploying a large language model on a terminal device is as follows: First, during the model loading phase, only the parameters of non-expert networks are loaded; the parameters of all expert networks are not loaded into memory during this phase. Second, during the model inference phase, other layers of non-expert networks perform inference normally. When the inference of the gated layer is completed and the index of the selected expert network for that layer is obtained, the system loads the parameters of that expert network from the hard drive into memory. After loading is complete, the inference of that expert network is then performed, while other unselected expert networks do not perform inference.

[0012] Preferably, the MoE structure is first a gating layer whose input is the output of the self-attention layer, and whose output controls the selection of multiple subsequent expert networks.

[0013] Preferably, all parameters are useful during inference. In the MoE, only the parameters of the gating layer and the expert network selected by the gating layer are activated, while the parameters of the unselected expert networks are not activated. The number of these inactive parameters is directly related to the total number of expert networks in each MoE layer.

[0014] Preferably, the larger the number of parameters in the Switch Transformer model, the greater the proportion of inactive parameters in the total number of parameters.

[0015] Preferably, the method sets up a global expert register that manages the expert network loaded from the disk. During inference, the expert network in the expert register is used directly for inference.

[0016] Preferably, all experts in this method are managed by an expert register, and the memory occupied by the parameters of the non-expert network and the memory occupied by the expert register constitute the total memory used in this invention.

[0017] Compared with existing technologies, this invention provides a method for deploying sparse large language models for terminal devices, which has the following advantages:

[0018] This deployment method for sparse large language models for terminal devices improves upon the Switch Transformer network structure through probabilistic statistics and parallelization techniques. In the Switch Transformer network structure, the FFN layer in the T5 model is replaced with a MoE structure. A global expert register is designed to manage the expert network loaded from disk. During inference, the expert network in the expert register is directly used for inference. This enables the deployment of models far exceeding the memory limit of terminal devices at the cost of a small increase in inference time, and achieves inference deployment with zero precision loss. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the network structure of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 A deployment method for sparse large language models for terminal devices is proposed. Based on the SwitchTransformer network structure, improvements are made through probabilistic statistics and parallelization techniques to deploy models far exceeding the memory limit of terminal devices at the cost of a small increase in inference time, and to achieve inference deployment with zero precision loss.

[0022] In the Switch Transformer network architecture, the FFN layer in the T5 model is replaced with a MoE structure. The MoE structure is first a gating layer, whose input is the output of the self-attention layer. The output of this layer controls the selection of multiple expert networks. The structure and parameter size of each expert network are the same as those of the FFN layer in T5. In traditional large language model deployment scenarios, before the model performs inference on a cloud server, all parameters of the Switch Transformer model are stored in memory. During inference, all parameters except the MoE layer are activated, meaning all parameters are useful during inference. In the MoE, only the parameters of the gating layer and the expert network selected by the gating layer are active, while the parameters of the unselected expert networks are not activated. The number of these inactive parameters is directly related to the total number of expert networks in each MoE layer. Therefore, the larger the number of parameters in the Switch Transformer model, the larger the proportion of inactive parameters in the total number of parameters. For example, when the number of experts in each MoE layer in the Switch Transformer is set to 128, the activated parameters account for only one percent of the total number of parameters. If only activation parameters are loaded on the terminal device, memory can be saved by a hundredfold, which is the source of this invention. Therefore, a very simple method can be obtained to deploy such large language models on terminal devices. First, during the model loading phase, only the parameters of non-expert networks are loaded; the parameters of all expert networks are not loaded into memory during this phase. Second, during the model inference phase, other layers of non-expert networks perform inference normally. When the gating layer finishes inference and obtains the selected expert network index, the system loads the parameters of that expert network from the hard drive into memory. After loading, inference is performed on that expert network, while other unselected expert networks do not perform inference. This simple method allows large language models to run within the memory limitations of terminal devices.

[0023] However, since the activation parameters during each inference are related to all departmental control layers, and the activation parameters are different each time, how to ensure the loading of activation parameters and minimize the negative impact of this method on the model's inference time is a consideration of this invention. By running the original model on a large dataset and statistically analyzing the expert selection order of all departmental control layers during a single inference, and arranging them sequentially, we found that for a single inference, once the experts selected by the preceding MoE layers are determined, the expert selection of subsequent layers becomes highly uneven. For example, after selecting expert number 1 in the first layer, the probability of selecting experts number 2 and 5 in the second layer reaches over 90%; when selecting expert number 1 in the first layer and expert number 2 in the second layer, the probability of selecting experts number 3 and 5 in the third layer reaches over 94%. Therefore, after determining the experts selected by the preceding layers, the experts selected by the subsequent layers can be predicted with a high success rate. This invention uses this method: after determining the experts of the previous layer, the experts of the next layer are predicted and pre-loaded, thus saving the time of loading experts in the next layer. This process is performed on a sub-thread, running in parallel with the computation and inference of the previous layer. Since computational inference is performed on the GPU, while the loading process is done from the hard drive, it is I / O intensive. Therefore, parallelizing the two processes does not affect the inference speed.

[0024] Finally, to ensure the smooth execution of the above method, a global expert register needs to be designed. The expert register manages the expert networks loaded from disk, and during inference, it directly uses the expert networks in the expert register. All experts in the method of this invention are managed by the expert register; the memory occupied by the parameters of non-expert networks and the memory occupied by the expert register constitute the total memory used by this invention. Figure 1 This is the network structure diagram of the invention. As shown in the diagram, when the gating layer selects an expert, it checks if the expert is matched in the expert register. If matched, calculation is performed directly; otherwise, the expert is loaded from the hard drive into the expert register and calculated. Simultaneously, another thread predicts which expert the next layer will select and pre-loads it into the expert register. N in the diagram represents the number of repetitions for this module, the same as in the T5 model. This invention achieves the deployment of a large language model within the memory limitations of the terminal device using this method.

[0025] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for deploying a sparse large language model for terminal devices, characterized in that: This method improves upon the SwitchTransformer network structure through probabilistic statistics and parallelization techniques. It enables the deployment of models far exceeding the memory limits of terminal devices with only a slight increase in inference time, achieving inference deployment with zero precision loss. In the Switch Transformer network structure, the FFN layer in the T5 model is replaced with a MoE structure. Before inference on the cloud server, all parameters of the Switch Transformer model are stored in memory. During inference, all parameters of all layers except the MoE layer are activated. Deploying a large language model on the terminal device is as follows: First, during the model loading phase, only the parameters of non-expert networks are loaded; all expert network parameters are not loaded into memory during this phase. Second, during the model inference phase, other layers of non-expert networks perform inference normally. This method uses a global expert register to manage the expert networks loaded from disk. After the expert selected for the previous network layer is determined, probabilistic statistics are used to predict the expert for the next layer and preload it. This preloading process is performed on a sub-thread, running in parallel with the computation and inference of the previous layer. Probabilistic statistics refer to the characteristic that once the expert selected for the previous MoE layer is determined, the expert selection for the next layer becomes highly uneven. The system predicts the expert selected for the next layer based on the determined experts for the previous layer. When the inference of the current gating layer is completed and the selected expert network number is obtained, the system checks if it matches in the expert register. If it matches, the system directly uses the expert network in the register for computation and inference; otherwise, the system loads the parameters of the selected expert network from the hard drive into the register and performs computation. Other unselected expert networks are not used for inference.

2. The deployment method of a sparse large language model for terminal devices according to claim 1, characterized in that: The MoE structure is first and foremost a gated layer whose input is the output of the self-attention layer. The output of this layer controls the selection of multiple expert networks that follow.

3. The deployment method of a sparse large language model for terminal devices according to claim 1, characterized in that: All parameters are useful during inference. In MoE, only the parameters of the gating layer and the expert network selected by the gating layer are activated, while the parameters of the unselected expert networks are not activated. The number of these unactivated parameters is directly related to the total number of expert networks in each MoE layer.

4. The deployment method of a sparse large language model for terminal devices according to claim 1, characterized in that: The larger the number of parameters in a Switch Transformer model, the greater the proportion of inactive parameters in the total number of parameters.

5. The deployment method of a sparse large language model for terminal devices according to claim 1, characterized in that: In this method, all experts are managed by the expert register. The memory occupied by the parameters of the non-expert network and the memory occupied by the expert register constitute the total memory used by this method.

Images