Large model acceleration method based on threshold skip

Abstract

The invention relates to a large model acceleration method, in particular to a large model acceleration method based on threshold skip. According to the method, the considerable effect is taken into account while the efficiency and benefits are integrated. The method aims at solving the problems that although existing large model acceleration methods based on layer skipping can obtain good efficiency benefits, the layer skipping operation is neglected so that the effect can not be accepted, and specifically, the smoothness of a text generated by a large model is not high, and the accuracy rate is not high. According to the method, a threshold-based layer-skipping algorithm is provided, and compared with a traditional layer-skipping algorithm, more excellent effects can be obtained while the efficiency and benefits are further improved. Specifically, lexical element input features of a certain middle layer of each model need to be scored based on a threshold-value layer-skipping algorithm, when the score exceeds a preset threshold value, the lexical element needs to participate in the calculation of the current layer, otherwise, the calculation of the current layer can be directly skipped. By means of the design, layer skipping operation can be more expanded, efficiency and benefits can be further improved, and the effect can be kept within an acceptable range.

Description

Technical Field

[0001] The invention relates to a large model acceleration method based on threshold skipping, belonging to the technical field of model acceleration in natural language processing. Background Art

[0002] Model acceleration refers to the optimization of the model's reasoning speed, which is mainly divided into acceleration of the pre-filling stage and acceleration of the decoding stage. It is the basis of model reasoning and is of great value to many natural language processing and information retrieval applications.

[0003] Previous model acceleration methods often only consider the acceleration of the pre-filling stage or the decoding stage. The large model acceleration method based on skip layers is a representative solution that can accelerate both the pre-filling stage and the decoding stage. Although the model's reasoning efficiency can be greatly improved under the large model acceleration method based on skip layers, the model's effect may be seriously affected, which is specifically reflected in the low reasoning accuracy and the unsmooth generated text. Summary of the invention

[0004] The purpose of the present invention is to innovatively propose a large model acceleration method based on threshold skipping, in order to address the technical problem that the existing large model acceleration method based on skipping can achieve a significant improvement in reasoning efficiency but is limited to poor results.

[0005] The essence of skipping layers is that for each layer in the model calculation, a certain proportion of the input of the current layer is determined to participate in the calculation, and the remaining input is regarded as skipping the calculation of the current layer.

[0006] The innovation of the present invention is that: for the first time, a large model based on threshold layer skipping is proposed, and the threshold is used for appropriate layer skipping. Different from the previous layer skipping method, this method can more freely control the proportion of skipping the current layer calculation of the model when skipping. Because the previous method fixed the input proportion of the current layer calculation to be skipped at each layer, it is likely to cause unnecessary calculations to be redundantly retained, while the required calculations are inappropriately skipped. The threshold layer skipping method adopted by this method significantly improves the effect of the model while retaining the efficiency improvement of the large model acceleration method of layer skipping.

[0007] The present invention is implemented by adopting the following technical solutions.

[0008] A large model acceleration method based on threshold skipping layer includes the following steps:

[0009] Step 1: Get the large model to be accelerated and a piece of input text segmented into words;

[0010] Specifically, the large model can be divided into L layers, and the input text will be initially converted into input features X 0 .

[0011] Step 2: Input features X 0 Enter the calculation of the previous l layers and output feature X l ;

[0012] Specifically, in the calculation of the first l layers, each word may also have skipped layers, that is, the actual calculation may not have l layers.

[0013] Step 3: Feature X l Input the calculation of the next layer, i.e., layer l+1, and score the input first;

[0014] Specifically, if the weight of the scorer at layer l+1 is W l+1 , then the scoring process is expressed as:

[0015] S l+1 =sigmoid(X l W l+1 )

[0016] Among them, sigmoid is a nonlinear monotonically increasing function that can limit the input to any value between 0 and 1.

[0017] Step 4: For each score S l+1 , determine whether it is greater than or equal to the defined threshold;

[0018] Specifically, if the threshold is defined as p, the judgment process is:

[0019] M l+1 =indicator(S l+1 ≥p)

[0020] Among them, indicator is a Boolean function. If the input condition is met, its output is 1, otherwise it outputs 0.

[0021] Step 5: For values ​​greater than or equal to the defined threshold, the corresponding M l+1 For features with a value of 1, the corresponding calculation is performed; for features with a value less than the defined threshold, that is, M l+1 If the feature is 0, the current calculation is skipped directly;

[0022] Specifically, we first need to l+1 , separate the input features that need to be calculated from the features that do not need to be calculated. The process can be expressed as:

[0023]

[0024] Wherein, split is a feature segmentation function, which separates the features according to the Boolean values ​​(0 or 1) corresponding to the features one by one; further, step 5 also includes the following steps:

[0025] Step 5.1: Input features to be calculated Calculate and output features

[0026] Specifically, the feature calculation process is as follows:

[0027]

[0028] Among them, transformer is a layer of calculation parameters and processes of the large model.

[0029] Step 5.2: Input features that do not require calculation Skip and output features

[0030] Specifically, the process of skipping features is as follows:

[0031]

[0032] We don't need to do anything during the skip process.

[0033] Step 6: Merge the output features corresponding to the calculated features and the uncomputed output features As the final output X l+1 , which is the input of the next layer;

[0034] Specifically, the output merging process is as follows:

[0035]

[0036] Among them, concat is one to two outputs The function for concatenation is, necessarily, sort, a function that restores the order of the output according to the input order of the features, because the order has been disrupted by the split function in step 5.

[0037] Step 7: Use the output features of layer l+1 as the input features of the subsequent layer L-l+1 to calculate and output the final features X L ;

[0038] Specifically, in the calculation of the subsequent L-l+1 layers, each word may also have a skipped layer, that is, the actual calculation may not have the L-l+1 layer.

[0039] Beneficial Effects

[0040] The method of the present invention combines a threshold-based layer skipping model and has the following advantages over the prior art:

[0041] 1. This method overcomes the drawback that in the traditional large model acceleration scheme based on skip layers, while achieving a significant improvement in acquisition efficiency, the effect is not satisfactory. It proposes a large model acceleration scheme based on threshold skip layers. Instead of using the previous skip layer method, this method uses the threshold skip layer method, thus achieving the optimal compromise between efficiency and effect.

[0042] 2. This method significantly improves efficiency and effect. Compared with large models that do not use this method, the large model based on threshold skip layers achieves a 1.2 - fold acceleration in latency and a 1.8 - fold reduction in video memory occupancy. In terms of effect, the accuracy of this method is improved by 4.5 percentage points compared with traditional methods on the publicly available dataset MMLU, and by 6.5 percentage points on the publicly available dataset CEval. This demonstrates the superiority of using threshold skip layers. Brief Description of the Drawings

[0043] Figure 1 It is a schematic diagram for a single layer to implement threshold skip layer; Detailed Implementation Manner

[0044] The present invention will be further described in detail below with reference to the drawings and embodiments.

[0045] Embodiment

[0046] A large model acceleration method based on threshold skip layers includes the following steps:

[0047] Step 1: Obtain a large model to be accelerated with 4 layers and an input text [acceleration, large, model] segmented into words;

[0048] Specifically, the input text will be initially converted into input features

[0049] Step 2: The input feature X 0 enters the calculation of the first 2 layers and outputs a feature

[0050] Specifically, in the calculation of the first 2 layers, each word may also skip some layers, that is, the actual calculation may not be 2 layers.

[0051] Step 3: The feature X 2 is input into the calculation of the next layer, i.e., the 3rd layer, and the input is scored first;

[0052] Specifically, if the weight of the scorer in the 3rd layer is W 3 , then the scoring process is expressed as:

[0053]

[0054] Among them, sigmoid is a nonlinear monotonically increasing function, which can limit the input to any value between 0 and 1; further assume that

[0055] Step 4: For each score S 3 , determine whether it is greater than or equal to the defined threshold;

[0056] Specifically, if the threshold is defined as p, and p = 0.5, the judgment process is:

[0057]

[0058] Among them, indicator is a Boolean function, which outputs 1 if the input condition is met, otherwise it outputs 0; that is,

[0059] Step 5: For values ​​greater than or equal to the defined threshold, the corresponding M 3 For features with a value of 1, the corresponding calculation is performed; for features with a value less than the defined threshold, that is, M 3 If the feature is 0, the current calculation is skipped directly;

[0060] Specifically, we first need to 3 , separate the input features that need to be calculated from the features that do not need to be calculated. The process can be expressed as:

[0061]

[0062] Wherein, split is a feature segmentation function, which separates the features according to the Boolean values ​​(0 or 1) corresponding to the features one by one; further, step 5 also includes the following steps:

[0063] Step 5.1: Input features to be calculated Calculate and output features

[0064] Specifically, the feature calculation process is as follows:

[0065]

[0066] Among them, transformer is a layer of calculation parameters and processes of the large model.

[0067] Step 5.2: Input features that do not require calculation Skip and output features

[0068] Specifically, the process of skipping features is as follows:

[0069]

[0070] We don't need to do anything during the skip process.

[0071] Step 6: Merge the output features corresponding to the calculated features and the uncomputed output features As the final output X 3 , which is the input of the next layer;

[0072] Specifically, the output merging process is as follows:

[0073]

[0074] Among them, concat is one to two outputs The function for concatenation is, necessarily, sort, a function that restores the order of the output according to the input order of the features, because the order has been disrupted by the split function in step 5.

[0075] Step 7: Use the output features of the third layer as the input features of the next layer to calculate and output the final features X 4 ;

[0076] Specifically, in the calculation of the subsequent 1 layer, each word may also have a skipped layer, that is, the actual calculation may not have 1 layer.

[0077] The above is only a preferred embodiment of the present invention, and the present invention should not be limited to the contents disclosed in the embodiment and the drawings. Any equivalent or modification completed without departing from the spirit disclosed in the present invention shall fall within the scope of protection of the present invention.

Claims

1. A large model acceleration method based on threshold skipping, characterized in that: The following steps are involved: Step 1: Get an L-layer large model to be accelerated and a piece of input text that is segmented into words. The input text will be initially converted into input features X 0 ; Step 2: Input features X 0 Enter the calculation of the previous l layers and output feature X l ; Step 3 : Feature X l Input the calculation of the next layer, i.e., layer l+1, and first score the input S l+1 ; Step 4: For each score S l+1 , determine whether it is greater than or equal to the defined threshold p, and obtain the binary score M l+1 ; Step 5: For values ​​greater than or equal to the defined threshold, the corresponding M l+1 For features with a value of 1, the corresponding calculation is performed; for features with a value less than the defined threshold, that is, M l+1 If the feature is 0, the current calculation is skipped directly; Step 6: Merge the output features corresponding to the calculated features and the uncomputed output features As the final output X l+1 , which is the input of the next layer; Step 7: Use the output features of layer l+1 as the input features of the subsequent layer L-l+1 to calculate and output the final features X L .

2. A large model acceleration method based on threshold skipping as claimed in claim 1, characterized in that: In step 1, the large model to be accelerated and a piece of input text segmented into words are obtained; Specifically, the large model can be divided into L layers, and the input text will be initially converted into input features X 0 .

3. A large model acceleration method based on threshold skipping as claimed in claim 1, characterized in that: In step 2, we need to input feature X 0 Enter the calculation of the previous l layers and output feature X l ; Specifically, in the calculation of the first l layers, each word may also have skipped layers, that is, the actual calculation may not have l layers.

4. A large model acceleration method based on threshold skipping as claimed in claim 1, characterized in that: In step 3, feature X l Input the calculation of the next layer, i.e., layer l+1, and score the input first; Specifically, if the weight of the scorer at layer l+1 is W l+1 , then the scoring process is expressed as: S l+1 =sigmoid(X l W l+1 ) Among them, sigmoid is a nonlinear monotonically increasing function that can limit the input to any value between 0 and 1.

5. A large model acceleration method based on threshold skipping as claimed in claim 1, characterized in that: Step 4: For each score S l+1 , determine whether it is greater than or equal to the defined threshold; Specifically, if the threshold is defined as p, the judgment process is: M l+1 =indicator(S l+1 ≥p) Among them, indicator is a Boolean function. If the input condition is met, its output is 1, otherwise it outputs 0.

6. A large model acceleration method based on threshold skipping as claimed in claim 1, characterized in that: Step 5 is greater than or equal to the defined threshold, that is, the corresponding M l+1 For features with a value of 1, the corresponding calculation is performed; for features with a value less than the defined threshold, that is, M l+1 If the feature is 0, the current calculation is skipped directly; Specifically, we first need to l+1 , separate the input features that need to be calculated from the features that do not need to be calculated. The process can be expressed as: Among them, split is a feature segmentation function, which separates the features according to the Boolean values ​​(0 or 1) corresponding to the features.

7. A large model acceleration method based on threshold skipping as claimed in claim 1, characterized in that: Step 5 should also include the following steps Step 5.1: Input features to be calculated Calculate and output features Step 5.2: Input features that do not require calculation Skip and output features 8. A large model acceleration method based on threshold skipping as claimed in claim 7, characterized in that: Step 5.1: Input features to be calculated Calculate and output features Specifically, the feature calculation process is as follows: Among them, transformer is a layer of calculation parameters and processes of the large model.

9. A large model acceleration method based on threshold skipping as claimed in claim 7, characterized in that: Step 5.2: For input features that do not need to be calculated Skip and output features Specifically, the process of skipping features is as follows: We don't need to do anything during the skip process.

10. A large model acceleration method based on threshold skipping as claimed in claim 1, characterized in that: Step 6: Merge the output features corresponding to the calculated features and the uncomputed output features As the final output X l +1 , which is the input of the next layer; Specifically, the output merging process is as follows: Among them, concat is one to two outputs The function for concatenation is, necessarily, sort, a function that restores the order of the output according to the input order of the features, because the order has been disrupted by the split function in step 5.

11. A large model acceleration method based on threshold skipping as claimed in claim 1, characterized in that: Step 7: Use the output features of layer l+1 as the input features of the subsequent layer L-l+1 to calculate the final features X L ; Specifically, in the calculation of the subsequent L-l+1 layers, each word may also have a skipped layer, that is, the actual calculation may not have the L-l+1 layer.

Images