Large model reasoning acceleration system and method based on adaptive layer skipping

Abstract

A large model reasoning acceleration system based on adaptive layer skipping comprises an off-line learning module, an on-line learning module and a batch processing module, the off-line learning module collects and integrates importance information of a pre-filling stage in previous reasoning results, and is used for off-line learning of a to-be-skipped sub-layer of each request in the pre-filling stage; the online learning module collects and integrates a small amount of content of the current request in the decoding stage, and updates a to-be-skipped sub-layer of each request in the decoding stage in real time; and the batch processing module optimizes and adjusts the sublayers to be skipped to realize simultaneous reasoning of a plurality of requests.

Description

Technical Field

[0001] The present invention relates to a technology in the field of large language models, specifically a large model reasoning acceleration system and method based on adaptive skipping layers. Background Art

[0002] The autoregressive nature of the Large Language Model (LLM) enables contextual learning during inference, but this results in a high cost for inference. The layer skipping strategy is expected to reduce inference costs and latency without modifying the model structure. The specific method is to skip layers that do not affect the inference results during inference. Existing methods include early skipping, periodic skipping, and early exit. However, these layer skipping strategies have certain limitations. Summary of the invention

[0003] The present invention aims at the shortcomings that the existing layer-skipping strategy needs additional fine-tuning, cannot adapt to personalized layer-skipping of different models and sequences, has difficulty in sequence batch processing, and needs to consume additional computing power to verify the guessed Token sequence. When the verification fails, it needs to roll back and recalculate, and it cannot save the KV Cache storage cost. A large model reasoning acceleration system and method based on adaptive layer-skipping is proposed, which can adaptively select the sublayer to be skipped and minimize the impact on the reasoning performance of the model. By efficiently learning the importance distribution in the historical reasoning process, a layer-skipping strategy for the prefilling phase can be constructed; by performing online importance learning in dynamic intermediate data, the layer-skipping decision can be further improved in the decoding phase. In addition, in the batch processing scenario, different sublayers are processed differently according to the importance distribution characteristics of different sublayers. Compared with the existing layer-skipping technology, the present invention is more personalized, greatly improves the degree of adaptation to different models and different sequences, and the text generation quality exceeds the existing layer-skipping technology in the test of multiple models.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a large model reasoning acceleration system based on adaptive skipping layer, comprising: an offline learning module, an online learning module and a batch processing module, wherein: the offline learning module collects and integrates the importance information of the pre-filling stage in the previous reasoning results, and is used for offline learning the sub-layers to be skipped in the pre-filling stage of each request; the online learning module collects and integrates a small amount of content of the current request in the decoding stage, and updates the sub-layers to be skipped in the decoding stage of each request in real time; the batch processing module realizes the simultaneous reasoning of multiple requests by optimizing and adjusting the sub-layers to be skipped.

[0006] The previous reasoning results refer to: using a large model to perform reasoning on unlabeled corpus and collecting layer input and output cosine similarity information during the reasoning process.

[0007] The sublayers mentioned refer to the attention sublayer and linear sublayer contained in a complete transformer layer, which are connected in series and contain residual structures inside.

[0008] The small amount of content of the current request in the decoding stage refers to: the current request is autoregressively decoded in the first 20 steps (20 steps are the length of the online learning window) of the decoding stage. This part of the content will be collected by the online learning module to decide which additional linear sub-layers to skip after 20 steps.

[0009] The offline learning module includes: an attention sublayer input and output collection unit, an attention sublayer compensation unit, a linear sublayer input and output collection unit, a linear sublayer compensation unit, and a merge sorting unit. Among them: the attention sublayer input and output collection unit will collect the attention sublayer input vector and output vector generated by the large model when performing offline reasoning on the unlabeled corpus, and calculate the cosine similarity of the input vector and the output vector; the attention sublayer compensation unit will calculate the average value of the module length ratio of the attention sublayer input vector and the output vector; the linear sublayer input and output collection unit will collect the linear sublayer input vector and output vector generated by the large model when performing offline reasoning on the unlabeled corpus, and calculate the cosine similarity of the input vector and the output vector; the linear sublayer compensation unit will calculate the average value of the module length ratio of the linear sublayer input vector and the output vector; the merge sorting unit will comprehensively sort the cosine similarity of the input and output vectors of the attention sublayer and the linear sublayer, select the sublayer with the highest input and output cosine similarity, and skip it first.

[0010] The online learning module includes: an attention sub-layer input and output collection unit, an attention sub-layer compensation unit, a linear sub-layer input and output collection unit, a linear sub-layer compensation unit and an additional skip layer decision unit. Among them: the attention sublayer input and output collection unit will collect the attention sublayer input vectors and output vectors generated in the first 20 steps of the decoding stage that have not been skipped yet, and calculate the cosine similarity of the input vector and the output vector; the attention sublayer compensation unit will calculate the average value of the module length ratio of the attention sublayer input vector and the output vector generated in the first 20 steps of the decoding stage that have not been skipped yet; the linear sublayer input and output collection unit will collect the linear sublayer input vectors and output vectors generated in the first 20 steps of the decoding stage that have not been skipped yet, and calculate the cosine similarity of the input vector and the output vector; the linear sublayer compensation unit will calculate the average value of the module length ratio of the linear sublayer input vector and the output vector generated in the first 20 steps of the decoding stage that have not been skipped yet; the additional skip layer decision unit will compare the cosine similarity of the input and output vectors of the sublayers that have not been skipped yet with the similarity statistically calculated in the offline stage, and additionally skip those sublayers whose similarity exceeds the lowest similarity in the layers that have been decided to be skipped in the offline stage.

[0011] When the batch processing module encounters multiple requests for different layer hopping solutions during the batch processing process, it uses the principle of minority obeys majority to generate the final layer hopping solution using the customized GPU kernel function written in CUDA.

[0012] The present invention relates to a large model reasoning acceleration method based on adaptive layer skipping based on the above system. When a new model is used for reasoning, the first N requests are fully executed without layer skipping, the importance of each sub-layer therein is calculated, and the sub-layer to be skipped is screened out based on this, and a compensation scheme for the sub-layer to be skipped is calculated, which is used to compensate when the sub-layer is skipped; during online decoding, the first P tokens are fully executed without layer skipping, the importance of each sub-layer therein is calculated, the sub-layer to be skipped is screened out through a threshold, and a compensation scheme for the sub-layer to be skipped is calculated, which is used to compensate when the sub-layer is skipped; when multiple requests determine the sub-layer to be skipped, the throughput of the model is improved by merging the execution of multiple requests, and when each request has an attention sub-layer or a linear sub-layer to be skipped as a conflicting layer, a customized GPU kernel function is used to implement partial request execution or the principle of minority obeys majority is used to determine whether to skip the sub-layer. Technical Effects

[0013] The present invention obtains the unimportant sublayers to be skipped in the pre-filling stage and the decoding stage of each request through an offline learning module and an online learning module, and allows multiple requests to be executed simultaneously through a batch processing module. The attention sublayer and the linear sublayer are considered separately, rather than considering the transformer layer as a whole, making the layer skipping strategy more flexible. The offline learning module determines the layer skipping strategy of the request in the pre-filling stage, and the online learning module determines the layer skipping strategy of the request in the decoding stage. Compared with the prior art, the present invention does not require additional computing power to fine-tune the model, can realize batch execution of requests, and various accuracy indicators are better than the current layer skipping model under the same acceleration ratio. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of the present invention;

[0015] Figure 2 It is a schematic diagram of layer jumping in two stages of the present invention;

[0016] Figure 3 is the layer importance measurement indicator graph;

[0017] Figure 4 Schematic diagram of the impact of online learning window length;

[0018] In the figure: (a) is the attention sub-layer case; (b) is the linear sub-layer case. DETAILED DESCRIPTION

[0019] like Figure 1 As shown, this embodiment relates to a large model reasoning acceleration method based on adaptive layer skipping, including:

[0020] In the first step, the first N requests are fully executed without skipping layers, and the importance of each sub-layer is calculated. Based on this, the sub-layer to be skipped is screened out and a compensation scheme for the sub-layer to be skipped is calculated for compensation when the sub-layer is skipped.

[0021] like Figure 3 As shown, the importance of the input-output similarity Among them: a is the input vector of a sublayer, and b is the output vector.

[0022] The filtering out of the sublayers to be skipped refers to: calculating the layer importance statistics of the first N requests in the pre-filling stage, and skipping the m least important sublayers according to the speedup ratio and the importance statistics of each layer, specifically: the importance statistics of the jth layer The preset parameter m is used to balance the speedup ratio and accuracy. A higher m value will lead to a higher speedup ratio but also a higher accuracy loss. Assuming that the execution time of the attention sublayer and the linear sublayer is the same, the speedup ratio is calculated as L / (Lm), where L is the total number of sublayers in the original large model. However, since the execution time of the attention sublayer and the linear sublayer is different in actual execution, the actual speedup ratio needs to be tested.

[0023] The compensation scheme is calculated by the following method: the input-output gap of the first N tasks in the pre-filling stage is counted. After skipping the sub-layer, the input-output gap statistics are used to compensate the input so that the actual output is closer to the ideal output. Specifically, the input-output gap statistics of the jth layer are: Where: T i is the i-th reasoning task, |T i | is the prompt length of the task, a it and b it are the input and output vectors of the jth token in the i-th reasoning task at the jth layer.

[0024] In the second step, during the online decoding phase, the first P tokens are fully executed without skipping layers, the importance of each sub-layer is calculated, the sub-layer to be skipped is screened out through a threshold, and a compensation scheme for the sub-layer to be skipped is calculated to compensate for skipping the sub-layer.

[0025] The threshold screening means: according to the importance statistics of each layer of the first P tokens, the minimum similarity value in the skipped sub-layer is used as the threshold, and the layers exceeding the threshold are used as the sub-layers that need to be skipped in the current request decoding stage. Specifically, the statistical value of the jth layer is: Threshold Among them: a it and b it are the input and output vectors of the jth token in the i-th reasoning task at the jth layer.

[0026] In the third step, when multiple requests determine the sublayer to be skipped, the throughput of the model is improved by merging the execution of multiple requests. When each request has an attention sublayer or a linear sublayer to be skipped as a conflicting layer, a customized GPU kernel function is used to implement partial request execution or determine whether to skip the sublayer based on the principle of majority rule.

[0027] The conflicting layer means that a certain sublayer needs to be skipped for request a, but needs to be calculated (not skipped) for request b.

[0028] The customized GPU kernel function refers to: a kernel function written in CUDA code, which outputs a final 0 / 1 decision (0 for skipping, 1 for calculating) according to the majority rule based on the decision information of each request in a batch.

[0029] The principle of majority rule means that when more than half of the requests in a batch decide to skip a sublayer, the skipping is performed, otherwise the calculation is performed.

[0030] After specific experiments, the above method was run in the environment settings of Pytorch 2.0, CUDA 12.2, and NVDIA RTX 4090 GPU with a configuration of P=20. Multiple data sets were tested on multiple models, and multiple results achieved the best results at this stage, as follows:

[0031] As shown in the intermediate results of Table 1, the effect of the present invention in the pre-filling stage is demonstrated. When the number of target skipped sub-layers is the same, the performance of the present invention in both MMLU and SQuAD tasks is significantly better than other methods, thanks to the fact that the present invention adaptively determines the most similar sub-layers to be skipped. As the number of skipped sub-layers increases, the present invention still maintains a higher generation quality and continues to outperform other methods. In addition, the present invention has obvious advantages in accelerating reasoning. Because the present invention operates at the granularity of sub-layers and skips the more time-consuming attention sub-layers, the actual speedup ratio is better than other methods.

[0032] As shown in the results on the right side of Table 1, the effect of the present invention in the decoding stage is demonstrated. Please note that when a large number of sub-layers are skipped, the generation quality of other methods will drop significantly and lose usability. In contrast, the present invention still maintains relatively good generation quality, proving its robustness and effectiveness. In addition, the reasoning acceleration of the present invention in the decoding stage is also very obvious. Given the same generation quality, the present invention achieves greater reasoning acceleration than other methods.

[0033] Table 1

[0034] In order to prove the effectiveness of the present invention when the online learning window length P is set to 20, the hit rate of predicting layer importance under different P is explored. Figure 3 As shown in the figure, as P increases, the hit rate gradually improves. This improvement is because a longer window length allows recording features that are closer to the true features of the current context. When P is less than 20, the hit rate increases almost linearly with the window length; when P exceeds 20, the hit rate growth slows down, indicating that the optimal window length is about 20.

[0035] Compared with the prior art, this paper analyzes the limitations of the current layer-by-layer skipping strategy in accelerating inference execution, and proposes a novel adaptive, sub-layer-by-sub-layer skipping system and method. This invention does not require training and is applicable to the pre-filling and decoding stages. Through rigorous testing on various datasets and models, it is demonstrated that this invention surpasses the existing methods in terms of generation quality and inference performance.

[0036] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A large model reasoning acceleration system based on adaptive layer skipping, characterized in that: include: Offline learning module, online learning module and batch processing module, where: the offline learning module collects and integrates the importance information of the pre-filling stage in the previous reasoning results, and is used for offline learning of the sub-layers to be skipped in the pre-filling stage for each request; the online learning module collects and integrates a small amount of content in the decoding stage of the current request, and updates the sub-layers to be skipped in the decoding stage for each request in real time; the batch processing module realizes simultaneous reasoning of multiple requests by optimizing and adjusting the sub-layers to be skipped; The small amount of content of the current request in the decoding stage refers to: the first several steps of autoregressive decoding of the current request in the decoding stage to determine the subsequent additionally skipped linear sublayers; The importance of the input-output similarity Among them: a is the input vector of a sublayer, and b is the output vector.

2. The large model reasoning acceleration system based on adaptive layer skipping according to claim 1 is characterized in that: The offline learning module includes: an attention sublayer input and output collection unit, an attention sublayer compensation unit, a linear sublayer input and output collection unit, a linear sublayer compensation unit and a merge sorting unit, wherein: the attention sublayer input and output collection unit collects the attention sublayer input vector and output vector generated by the large model when performing offline reasoning on an unlabeled corpus, and calculates the cosine similarity of the input vector and the output vector; the attention sublayer compensation unit calculates the average value of the module length ratio of the attention sublayer input vector and the output vector; the linear sublayer input and output collection unit collects the linear sublayer input vector and output vector generated by the large model when performing offline reasoning on an unlabeled corpus, and calculates the cosine similarity of the input vector and the output vector; the linear sublayer compensation unit calculates the average value of the module length ratio of the linear sublayer input vector and the output vector; the merge sorting unit comprehensively sorts the cosine similarity of the input and output vectors of the attention sublayer and the linear sublayer, selects the sublayer with the highest input and output cosine similarity, and skips it first.

3. The large model reasoning acceleration system based on adaptive layer skipping according to claim 1 is characterized in that: The online learning module includes: an attention sublayer input and output collection unit, an attention sublayer compensation unit, a linear sublayer input and output collection unit, a linear sublayer compensation unit and an additional skip layer decision unit, wherein: the attention sublayer input and output collection unit collects the attention sublayer input vectors and output vectors that have not been skipped before the decoding stage, and calculates the cosine similarity of the input vector and the output vector; the attention sublayer compensation unit calculates the average value of the module length ratio of the attention sublayer input vector and the output vector that have not been skipped before the decoding stage; the linear sublayer compensation unit calculates the average value of the module length ratio of the attention sublayer input vector and the output vector that have not been skipped before the decoding stage; The layer input and output collection unit collects the linear sub-layer input vectors and output vectors generated by the large model several steps before the decoding stage that have not been skipped, and calculates the cosine similarity of the input vectors and output vectors; the linear sub-layer compensation unit calculates the average value of the module length ratio of the linear sub-layer input vectors and output vectors generated several steps before the decoding stage that have not been skipped; the additional skip layer decision unit compares the cosine similarity of the input and output vectors of the sub-layers that have not been skipped with the similarity statistically obtained in the offline stage, and additionally skips those sub-layers whose similarity exceeds the lowest similarity among the layers that have been decided to be skipped in the offline stage.

4. The large model reasoning acceleration system based on adaptive layer skipping according to claim 1 is characterized in that: When the batch processing module encounters multiple requests for different layer hopping solutions during the batch processing process, it uses the principle of minority obeys majority to generate the final layer hopping solution using the customized GPU kernel function written in CUDA.

5. A large model reasoning acceleration method based on adaptive layer skipping based on the system according to any one of claims 1 to 4, characterized in that: When using the new model for inference, the first N requests are fully executed without skipping layers, and the importance of each sub-layer is calculated. The sub-layers to be skipped are screened out based on this, and the compensation scheme for the sub-layers to be skipped is calculated for compensation when the sub-layer is skipped. During online decoding, the first P tokens are fully executed without skipping layers, and the importance of each sub-layer is calculated. The sub-layers to be skipped are screened out through a threshold, and the compensation scheme for the sub-layers to be skipped is calculated for compensation when the sub-layer is skipped. When multiple requests determine the sub-layers to be skipped, the throughput of the model is improved by merging the execution of multiple requests. When each request has an attention sub-layer or a linear sub-layer to be skipped as a conflicting layer, a customized GPU kernel function is used to implement partial request execution or to determine whether to skip the sub-layer based on the principle of minority obeys majority.

6. The large model reasoning acceleration method according to claim 5 is characterized in that: include: The first step is to completely execute the first N requests without skipping layers, calculate the importance of each sub-layer, and based on this, select the sub-layer to be skipped and calculate the compensation scheme for the sub-layer to be skipped, which is used to compensate when the sub-layer is skipped; The second step is to execute the first P tokens completely without skipping layers in the online decoding stage, calculate the importance of each sub-layer, filter out the sub-layers to be skipped by thresholds, and calculate the compensation scheme for the sub-layers to be skipped, which is used to compensate when skipping the sub-layers. In the third step, when multiple requests determine the sublayer to be skipped, the throughput of the model is improved by merging the execution of multiple requests. When each request has an attention sublayer or a linear sublayer to be skipped as a conflicting layer, a customized GPU kernel function is used to implement partial request execution or determine whether to skip the sublayer based on the principle of majority rule.

7. The large model reasoning acceleration method according to claim 6 is characterized in that: The filtering out of the sublayers to be skipped refers to: calculating the layer importance statistics of the first N requests in the pre-filling stage, and skipping the m least important sublayers according to the speedup ratio and the importance statistics of each layer, specifically: the importance statistics of the jth layer The preset parameter m is used to balance the speedup ratio and accuracy. A higher m value will bring a higher speedup ratio but also result in a higher accuracy loss. Assuming that the execution time of the attention sublayer and the linear sublayer is the same, the speedup ratio is calculated as L / (Lm), where L is the total number of sublayers in the original large model. However, since the execution time of the attention sublayer and the linear sublayer is different in actual execution, the actual speedup ratio needs to be tested.

8. The large model reasoning acceleration method according to claim 6 is characterized in that: The compensation scheme is calculated by the following method: the input-output gap of the first N tasks in the pre-filling stage is counted. After skipping the sub-layer, the input-output gap statistics are used to compensate the input so that the actual output is closer to the ideal output. Specifically, the input-output gap statistics of the jth layer are: Where: T i is the i-th reasoning task, |T i | is the prompt length of the task, a it and b it are the input and output vectors of the jth token in the i-th reasoning task at the jth layer.

9. The large model reasoning acceleration method according to claim 5 is characterized in that: The threshold screening means: according to the importance statistics of each layer of the first P tokens, the minimum similarity value in the skipped sub-layer is used as the threshold, and the layers exceeding the threshold are used as the sub-layers that need to be skipped in the current request decoding stage. Specifically, the statistical value of the jth layer is: Threshold Among them: a it and b it are the input and output vectors of the jth token in the i-th reasoning task at the jth layer.

10. The large model reasoning acceleration method according to claim 5 is characterized in that: The customized GPU kernel function refers to: a kernel function written in CUDA code, which outputs a final 0 / 1 decision (0 for skipping, 1 for calculating) according to the majority rule based on the decision information of each request in a batch.

Images