Method and system for rapidly improving activation sparsity of large language models

Abstract

The present invention provides a method and system for quickly improving the activation sparsity of a large language model, including: step 1: modifying the activation function of the feedforward layer of the large language model to a ReLU activation function; step 2: adding a ReLU activation function after the output of a gated linear unit module for enhancing nonlinear capabilities in the feedforward layer of the large language model; step 3: cleaning and mixing pre-trained corpora from multiple sources; step 4: selecting optimal training hyperparameters according to the hyperparameter scaling law; step 5: training the model on a 200B corpus, and adopting a distributed training strategy based on data parallelism, model parallelism, and pipeline parallelism during the training process; step 6: fine-tuning the model in instruction alignment and human-preferred data sets. By introducing a higher quality and diversified corpus mixture, the present invention not only maintains the sparsity of the model, but also improves the overall capability of the model, making it applicable to a wider range of application scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a method and system for rapidly improving activation sparsity of a large language model. Background Art

[0002] Large Language Models (LLMs) have significantly improved the quality of our daily lives and revolutionized our work environments with their superior text understanding and generation capabilities. Current state-of-the-art LLMs, such as GPT-4 and Claude-3, are typically hosted in data centers equipped with state-of-the-art GPUs (e.g., NVIDIA H100), which provide extensive high-bandwidth memory and have computational power reaching thousands of teraflops. However, the huge computational cost required for LLMs deployment and inference poses a considerable challenge to the wider application of LLMs.

[0003] To address this challenge, leveraging the built-in activation sparsity of large language models is a promising approach, as it improves inference efficiency by skipping useless computations by leveraging the zero elements in the output of the activation function of the large language model. However, recent large language models mainly tend to use more complex GELU and Swish activation functions, which lack the inherent sparse activation features.

[0004] Existing model activation sparsification methods also have serious limitations:

[0005] 1. Existing activation sparsification methods can only improve sparsity to a limited extent, up to 67%. Limited sparsity is unlikely to bring significant inference acceleration;

[0006] 2. The existing activation sparsification methods have the problem of model capability degradation that cannot be ignored. The best method currently has a 3%-5% capability degradation when the model is sparse. Summary of the invention

[0007] In view of the defects in the prior art, the object of the present invention is to provide a method and system for quickly improving the activation sparsity of a large language model.

[0008] The method for quickly improving the activation sparsity of a large language model provided by the present invention includes:

[0009] Step 1: Change the activation function of the feedforward layer of the large language model to the ReLU activation function to improve the sparsity of the model;

[0010] Step 2: Add a ReLU activation function after the output of the gated linear unit module used to enhance nonlinear capabilities in the feed-forward layer of the large language model to further improve the sparsity of the model;

[0011] Step 3: Clean and mix pre-training corpora from multiple sources to ensure high quality and diversity of the data;

[0012] Step 4: Select the optimal training hyperparameters, including learning rate and batch size, according to the hyperparameter scaling law to achieve the best model performance and sparsity;

[0013] Step 5: Train the model on a 200B corpus. During the training process, a distributed training strategy based on data parallelism, model parallelism, and pipeline parallelism is used to improve training efficiency, and mixed precision training technology is used to reduce the use of computing resources.

[0014] Step 6: Fine-tune the model on instruction-aligned and human-preferred datasets to improve the model’s interactive capabilities.

[0015] Preferably, the activation function expression of the feedforward layer of the large language model is:

[0016] FFN Swish (X,W up , W gate )=Swish(X·W gate )*(X·W up )

[0017] The ReLU activation function expression is:

[0018] FFN ReLU (X,W up , W gate )=ReLU(X·W gate )*(X·W up )

[0019] Among them, X represents the input, W up and W gat e denotes the weights in the feed-forward layer of the large language model.

[0020] Preferably, the expression of the learning rate is:

[0021] Learning Rate = A*C K

[0022] Among them, C is the computing resource, and A and K are the coefficients of the learning rate scaling law.

[0023] Preferably, the batch size maintains the scaling law of the original pre-trained model, expressed as:

[0024] Batch size = A*C K

[0025] Where C is the computing resource, and A and K are the coefficients of the batch size scaling law.

[0026] Preferably, step 3 includes: cleaning the pre-training corpus, removing preset low-quality data, duplicate data, and data with a relevance lower than a preset threshold, and mixing preset high-quality corpora from different sources, including web pages, codes, mathematics, and novels; and using natural language processing technology to segment, annotate, and filter the corpus.

[0027] The system for rapidly improving activation sparsity of a large language model provided by the present invention includes:

[0028] Module M1: Change the activation function of the feedforward layer of the large language model to the ReLU activation function to improve the sparsity of the model;

[0029] Module M2: Add a ReLU activation function after the output of the gated linear unit module used to enhance nonlinear capabilities in the feed-forward layer of the large language model, thereby further improving the sparsity of the model;

[0030] Module M3: Clean and mix pre-trained corpora from multiple sources, and use natural language processing technology to segment, annotate, and filter the corpora to ensure high quality and diversity of the data;

[0031] Module M4: Select optimal training hyperparameters, including learning rate and batch size, according to the hyperparameter scaling law to achieve the best model performance and sparsity;

[0032] Module M5: Train the model on a 200B corpus. During the training process, a distributed training strategy based on data parallelism, model parallelism, and pipeline parallelism is used to improve training efficiency, and mixed precision training technology is used to reduce the use of computing resources.

[0033] Module M6: Fine-tune the model in instruction-aligned and human-preferred datasets to improve the model's interactive capabilities.

[0034] Preferably, the activation function expression of the feedforward layer of the large language model is:

[0035] FFN Swish (X,W up , W gate )=Swish(X·W gate )*(X·W up )

[0036] The ReLU activation function expression is:

[0037] FFN ReLU (X,W up , W gate )=ReLU(X·Wgate )*(X·W up )

[0038] Among them, X represents the input, W up and W gate Both represent the weights in the feed-forward layer of the large language model.

[0039] Preferably, the expression of the learning rate is:

[0040] Learning Rate = A*C K

[0041] Among them, C is the computing resource, and A and K are the coefficients of the learning rate scaling law.

[0042] Preferably, the batch size maintains the scaling law of the original pre-trained model, expressed as:

[0043] Batch size = A*C K

[0044] Where C is the computing resource, and A and K are the coefficients of the batch size scaling law.

[0045] Preferably, the module M3 includes: cleaning the pre-training corpus, removing preset low-quality data, duplicate data, and data with a relevance lower than a preset threshold, and mixing preset high-quality corpora from different sources, including web pages, codes, mathematics, and novels; and using natural language processing technology to segment, annotate, and filter the corpus.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. Higher model sparsity ratio: By optimizing the model algorithm structure, the present invention improves the sparsity of the model to 91%, compared with the 67% that the existing technology can only achieve, significantly reducing the computing resources required for reasoning;

[0048] 2. Enhanced model capability: The existing technology often leads to a decrease in model capability while improving sparsity. However, the present invention not only maintains the sparsity of the model but also improves the overall capability of the model by introducing a higher quality and diversified corpus mixture, making it applicable to a wider range of application scenarios;

[0049] 3. Reduced training costs: By optimizing hyperparameters and improving corpus configuration during training, the present invention only requires 2% of the corpus of the original model to complete training, which greatly reduces the training cost compared to the original requirement of tens of trillions of tokens.

[0050] 4. Greater versatility: The present invention breaks through the limitation that the prior art can only be used for dense models, and is applicable to dense models including but not limited to LLaMA structure and Mistral structure, as well as mixed expert models such as Mixtral and Deepseek. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0052] Figure 1 The present invention is a flowchart of a training method for rapidly improving the sparsity of a large language model. DETAILED DESCRIPTION

[0053] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0054] Example 1

[0055] like Figure 1 As shown, the present invention provides a training method for quickly improving the sparsity of a large language model, comprising the following steps:

[0056] In step 101, the feedforward layer activation structure of the trained large language model is:

[0057] FFN Swish (X,W up , W gate )=Swish(X·W gate )*(X·W up )

[0058] Directly replace with ReLU activation function:

[0059] FFN ReLU (X,W up , W gate )=ReLU(X·W gate )*(X·W up )

[0060] This modification aims to exploit the intrinsic activation sparsity introduced by the ReLU activation function:

[0061] FFN ReLu (X,W up , W gate )=ReLU(X·W gate)*ReLU(X·W up )

[0062] Thereby improving the sparsity of the model, and then executing step 102;

[0063] In step 102, a ReLU module is further introduced into the gated linear unit module. By adding a ReLU activation function after the output of the gated linear unit module used to enhance the nonlinear capability in the feed-forward layer of the large language model, the sparsity of the model can be further enhanced, thereby reducing the consumption of computing resources, and then executing step 103;

[0064] In step 103, pre-trained corpora from multiple sources are cleaned and mixed. These corpora include but are not limited to web pages, codes, mathematics, novels, etc. The cleaning process includes removing low-quality data, duplicate data, and irrelevant data, and using natural language processing technology to segment, annotate, and filter the corpora to ensure high quality and diversity of the data, and then executing step 104;

[0065] In step 104, the optimal training hyperparameters are selected according to the hyperparameter scaling law. The selection process includes adjusting the learning rate, batch size, etc. to achieve the best model performance and sparsity. During the training process, since the original language model has embedded a lot of knowledge, the learning rate is 10% of the general large model learning rate. The expression of the learning rate Learning Rate is:

[0066] Learning Rate = A*C K

[0067] Among them, C is the computing resource, and A and K are the coefficients of the learning rate scaling law.

[0068] Substitute the specific parameters as follows:

[0069] Learning Rate = 0.1 * 0.3118 * C -0.125

[0070] Where C is the computing resource, and 0.3118 is the coefficient of the learning rate scaling law. The batch size maintains the scaling law of the original pre-trained model, expressed as:

[0071] Batch size = A*C K

[0072] Where C is the computing resource, and A and K are the coefficients of the batch size scaling law.

[0073] Substitute the specific parameters as follows:

[0074] Batch size = 0.2920*C 0.3271

[0075] Then execute step 105;

[0076] In step 105, the model is trained on a 200B corpus. During the training process, a distributed training strategy based on data parallelism, model parallelism, and pipeline parallelism is adopted. Since the training tokens are only 2% of the pre-trained ones, this solution mainly uses basic data parallelism. For models above 100B, the optimal parallelism of model parallelism and pipeline parallelism is searched through a small-scale hyperparameter grid to improve training efficiency, and mixed precision training technology is used to reduce the use of computing resources, and then step 106 is executed;

[0077] In step 106, the model is further fine-tuned in instruction alignment and human-preferred datasets. In order to ensure the model's security and instruction-compliance capabilities, the fine-tuning process uses full-parameter fine-tuning, that is, the trained sparse model is further trained on dialogue corpus and security corpus to enhance the model's interactive capabilities.

[0078] Example 2

[0079] The present invention also provides a system for rapidly improving the activation sparsity of a large language model. The system for rapidly improving the activation sparsity of a large language model can be implemented by executing the process steps of the method for rapidly improving the activation sparsity of a large language model, that is, those skilled in the art can understand the method for rapidly improving the activation sparsity of a large language model as a preferred implementation of the system for rapidly improving the activation sparsity of a large language model.

[0080] The system for rapidly improving the activation sparsity of a large language model provided by the present invention comprises: module M1: modifying the activation function of the feedforward layer of the large language model to a ReLU activation function, thereby improving the sparsity of the model; module M2: adding a ReLU activation function after the output of a gated linear unit module for enhancing nonlinear capabilities in the feedforward layer of the large language model, thereby further improving the sparsity of the model; module M3: cleaning and mixing pre-training corpora from multiple sources, and using natural language processing technology to segment, annotate and filter the corpora to ensure high quality and diversity of the data; module M4: selecting optimal training hyperparameters according to the hyperparameter scaling law, including learning rate and batch size, to achieve optimal model performance and sparsity; module M5: training the model on a 200B corpus, and adopting a distributed training strategy based on data parallelism, model parallelism and pipeline parallelism during the training process to improve training efficiency, and using mixed precision training technology to reduce the use of computing resources; module M6: fine-tuning the model in instruction alignment and human-preferred data sets, thereby improving the interactive ability of the model.

[0081] The activation function expression of the feedforward layer of the large language model is:

[0082] FFN Swish (X,W up , W gate )=Swish(X·W gate )*(X·W up )

[0083] The ReLU activation function expression is:

[0084] FFN ReLU (X,W up , W gate )=ReLU(X·W gate) *(X·W up )

[0085] Among them, X represents the input, W up and W gate Both represent the weights in the feed-forward layer of the large language model.

[0086] The expression of learning rate is:

[0087] Learning Rate = A*C K

[0088] Among them, C is the computing resource, and A and K are the coefficients of the learning rate scaling law.

[0089] Batch size maintains the scaling law of the original pre-trained model, expressed as:

[0090] Batch size = A*C K

[0091] Where C is the computing resource, and A and K are the coefficients of the batch size scaling law.

[0092] The module M3 includes: cleaning the pre-training corpus, removing preset low-quality data, duplicate data, and data with a relevance below a preset threshold, and mixing preset high-quality corpora from different sources, including web pages, codes, mathematics, and novels; and using natural language processing technology to segment, annotate, and filter the corpus.

[0093] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0094] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for rapidly improving activation sparsity of a large language model, characterized in that: include: Step 1: Change the activation function of the feedforward layer of the large language model to the ReLU activation function to improve the sparsity of the model; Step 2: Add a ReLU activation function after the output of the gated linear unit module used to enhance nonlinear capabilities in the feed-forward layer of the large language model to further improve the sparsity of the model; Step 3: Clean and mix pre-training corpora from multiple sources to ensure high quality and diversity of the data; Step 4: Select the optimal training hyperparameters, including learning rate and batch size, according to the hyperparameter scaling law to achieve the best model performance and sparsity; Step 5: Train the model on a 200B corpus. During the training process, a distributed training strategy based on data parallelism, model parallelism, and pipeline parallelism is used to improve training efficiency, and mixed precision training technology is used to reduce the use of computing resources. Step 6: Fine-tune the model in instruction-aligned and human-preferred datasets to improve the model’s interactive capabilities; The step 3 includes: cleaning the pre-training corpus, removing preset low-quality data, duplicate data, and data with a relevance below a preset threshold, and mixing preset high-quality corpora from different sources, including web pages, codes, mathematics, and novels; and using natural language processing technology to segment, annotate, and filter the corpus.

2. The method for rapidly improving activation sparsity of a large language model according to claim 1, characterized in that: The activation function expression of the feedforward layer of the large language model is: FFN Swish (X,W up ,W gate )=Swish(X·W gate )*(X·W up ) The ReLU activation function expression is: FFN ReLU (X, W up ,W gate )=ReLU(X·W gate )*(X·W up ) Among them, X represents the input, W up and W gate Both represent the weights in the feed-forward layer of the large language model.

3. The method for rapidly improving activation sparsity of a large language model according to claim 1, characterized in that: The expression of learning rate is: Learning Rate=A*C K Among them, C is the computing resource, and A and K are the coefficients of the learning rate scaling law.

4. The method for rapidly improving activation sparsity of a large language model according to claim 1, characterized in that: Batch size maintains the scaling law of the original pre-trained model, expressed as: Batch size=A*C K Where C is the computing resource, and A and K are the coefficients of the batch size scaling law.

5. A system for rapidly improving activation sparsity of large language models, characterized in that: include: Module M1: Change the activation function of the feedforward layer of the large language model to the ReLU activation function to improve the sparsity of the model; Module M2: Add a ReLU activation function after the output of the gated linear unit module used to enhance nonlinear capabilities in the feed-forward layer of the large language model, thereby further improving the sparsity of the model; Module M3: Clean and mix pre-trained corpora from multiple sources, and use natural language processing technology to segment, annotate, and filter the corpora to ensure high quality and diversity of the data; Module M4: Select optimal training hyperparameters, including learning rate and batch size, according to the hyperparameter scaling law to achieve the best model performance and sparsity; Module M5: Train the model on a 200B corpus. During the training process, a distributed training strategy based on data parallelism, model parallelism, and pipeline parallelism is used to improve training efficiency, and mixed precision training technology is used to reduce the use of computing resources. Module M6: Fine-tune the model in instruction alignment and human-preferred datasets to improve the model's interactive capabilities; The module M3 includes: cleaning the pre-training corpus, removing preset low-quality data, duplicate data, and data with a relevance below a preset threshold, and mixing preset high-quality corpora from different sources, including web pages, codes, mathematics, and novels; and using natural language processing technology to segment, annotate, and filter the corpus.

6. The system for rapidly improving activation sparsity of a large language model according to claim 5, characterized in that: The activation function expression of the feedforward layer of the large language model is: FFN Swish (X,W up ,W gate )=Swish(X·W gate )*(X·W up ) The ReLU activation function expression is: FFN ReLU (X, W up ,W gate )=ReLU(X·W gate )*(X·W up ) Among them, X represents the input, W up and W gate Both represent the weights in the feed-forward layer of the large language model.

7. The system for rapidly improving activation sparsity of a large language model according to claim 5, characterized in that: The expression of learning rate is: Learning Rate=A*C K Among them, C is the computing resource, and A and K are the coefficients of the learning rate scaling law.

8. The system for rapidly improving activation sparsity of a large language model according to claim 5, characterized in that: Batch size maintains the scaling law of the original pre-trained model, expressed as: Batch size=A*C K Where C is the computing resource, and A and K are the coefficients of the batch size scaling law.

Images