Transform calculation optimization method based on mixed depth

Abstract

The invention relates to a mixed depth-based Transform calculation optimization method, which comprises the following steps of: setting a dynamic calculation budget, and setting a calculation budget threshold value for each Transform block; the dynamic importance weight of each token is calculated through a router module, a Top-K mechanism is used for selecting the token with the high weight to enter a self-attention calculation path, and the remaining tokens enter a residual path; the token with higher importance participates in calculation of a complete self-attention mechanism and an MLP layer, and the token with lower importance bypasses the calculation and directly enters the next layer through a residual path; the two are combined during final output; dynamically adjusting a calculation budget and a routing strategy according to calculation requirements of different stages; and the Transform model is applied to the processing of the long sequence task. According to the method, the problems of computing resource waste and low efficiency when the existing Transform model processes the long text sequence are solved, and flexible allocation of the computing budget is realized, so that unnecessary computing overhead is reduced, and the overall performance of the model is kept.

Description

Technical Field

[0001] The present invention relates to a natural language processing technology, and in particular to a transformer calculation optimization method based on hybrid depth. Background Art

[0002] In the field of natural language processing, the Transformer model is one of the most advanced models. However, traditional Transformer models usually perform the same amount of calculations on each input token (the basic unit of a text fragment, which can usually be a word, character, or subword), regardless of the importance of the token. This uniform distribution of calculations leads to a waste of computing resources, especially when processing long sequences, resulting in inefficiency and high inference costs.

[0003] The main shortcomings of existing Transformer models include:

[0004] 1. Waste of computational resources: All tokens receive the same computational processing, regardless of their importance in the task.

[0005] 2. Lack of dynamic adjustment: The amount of computing cannot be dynamically adjusted according to the specific needs of the token, resulting in low resource utilization efficiency.

[0006] Existing hybrid expert models optimize the use of computing resources by introducing multiple expert modules, but these models still face the problem of uneven distribution of computing resources. In addition, MoE models still perform the same amount of calculations in each module and cannot dynamically adjust the computing budget. Summary of the invention

[0007] The present invention aims to solve the problems of waste of computing resources and low efficiency of the existing Transformer model, especially when processing long sequences. This uniform computing distribution leads to unnecessary computing overhead and resource waste, and cannot flexibly cope with the importance differences of different tokens. A transformer computing optimization method based on hybrid depth is proposed.

[0008] The technical solution of the present invention is:

[0009] A hybrid depth-based transformer computing optimization method optimizes the computing efficiency of the Transformer model by introducing a dynamic computing allocation mechanism. The improvement of computing efficiency is mainly achieved through a dynamic computing resource allocation mechanism. This method dynamically weighs the importance of different tokens and flexibly allocates computing budgets, thereby reducing unnecessary computing overhead and maintaining the overall performance of the model. The specific optimization process is as follows:

[0010] Step 1: Dynamic calculation budget setting: A dynamic calculation budget setting method, each Transformer block sets a calculation budget threshold B, which indicates the total number of tokens allowed to participate in the self-attention mechanism and multi-layer perceptron calculation at the current stage;

[0011] Step 2: Introduction of the router module: The router module is the core component of the optimization solution, responsible for dynamically allocating computing resources in each Transformer computing block. The router module calculates the dynamic importance weight of each token, and uses the Top-K mechanism to select tokens with high weights to enter the self-attention calculation path. The remaining tokens will enter the residual path.

[0012] Step 3: Hybrid deep computation path design: dynamically allocate computation paths based on the importance of tokens; tokens with higher importance will participate in the full self-attention mechanism and MLP layer computations, while tokens with lower importance will bypass these computations and directly enter the next layer through the residual path; the two are merged at the final output;

[0013] Step 4: Dynamic calculation adjustment mechanism: During the model training and reasoning process, the calculation budget and routing strategy are dynamically adjusted according to the calculation requirements at different stages. By gradually increasing the value of Top-K, only a small number of important tokens are involved in the calculation in the early stage, and as the training progresses, more tokens will be gradually introduced into the calculation path.

[0014] Step 5: Application: Apply the Transformer model to process long sequence tasks.

[0015] Further, the specific steps are as follows:

[0016] Step 1: Dynamically calculate budget settings. The specific steps are as follows:

[0017] Step 1.1. Each Transformer block sets a computational budget threshold B, which indicates the total number of tokens allowed to participate in the self-attention mechanism and multi-layer perceptron calculations at the current stage; this threshold is an adjustable hyperparameter and is set according to task requirements;

[0018] Step 1.2: In the initial stage, the model calculates the initial importance weight of each token, sorts the tokens according to the weight value, selects the tokens with higher weights to enter the calculation path, and the remaining low-weight tokens will temporarily skip the calculation of the current block and enter the residual path;

[0019] Step 2: Introduction of the router module. The specific workflow is as follows:

[0020] Step 2.1. In each Transformer block, the dynamic importance weight w of each token is calculated through the router module. t , w t The weight calculation is done through linear projection and feedforward network; the formula is as follows:

[0021] w t =W·t i +b

[0022] Among them, t i is the embedding representation of token, W is the weight matrix of linear projection, and b is the bias term;

[0023] Step 2.3: Based on the calculated weights, the Top-K mechanism is used to select the first K tokens with the highest weights to enter the self-attention calculation path. The remaining tokens will enter the residual path to avoid unnecessary calculation overhead. The specific Top-K selection mechanism is as follows:

[0024] K=TopK(w t ,k)

[0025] Among them, TopK means selecting the first k elements with the highest weight value, k is the user-defined capacity value, which indicates the maximum number of tokens processed in each calculation block, and w t is the importance weight;

[0026] Step 3: Design of hybrid depth calculation path. The specific steps are as follows:

[0027] Step 3.1: After passing through the router module, each token enters a different calculation path; tokens with higher importance will participate in the calculation of the complete self-attention mechanism and MLP layer. The path is as follows:

[0028] f(t i )=SelfAttention(t i )+MLP(t i )

[0029] Among them, f(t i ) represents the calculation path function, SelfAttention represents the self-attention mechanism operation, and MLP represents the multi-layer perceptron operation;

[0030] Step 3.2: Tokens with lower importance will bypass these calculations and go directly to the next layer through the residual path; the formula is as follows:

[0031] r(t i )=Residual(t i )

[0032] Among them, r(t i ) represents the residual path function, and Residual represents the residual connection operation;

[0033] Step 3.3: At the final output, the tokens of the residual path and the tokens of the calculation path will be merged to ensure that the model can retain the global information of all tokens, but reduce unnecessary calculations;

[0034] Step 4: Dynamic calculation adjustment mechanism: During the model training and reasoning process, the calculation budget and routing strategy are dynamically adjusted according to the calculation requirements at different stages. By gradually increasing the Top-K value, only a small number of important tokens are involved in the calculation in the early stage, and as the training progresses, more tokens will be gradually introduced into the calculation path. This dynamic adjustment mechanism ensures that the model can maximize the calculation efficiency without losing performance.

[0035] Step 5: Practical application and performance improvement: Applying the Transformer model to process long sequence tasks can improve computational efficiency.

[0036] The beneficial effects of the present invention are:

[0037] The traditional Transformer model has the problem of wasting computing resources and low efficiency when processing long text sequences, especially when the sequence contains a large number of unimportant tokens. To solve this problem, the present invention proposes a dynamic computing allocation mechanism that can dynamically adjust the computing budget according to the importance of the token. Through this method, the computing overhead of the model in long sequence processing can be reduced and the overall performance stability can be maintained.

[0038] In NLP tasks, especially in machine translation, text summarization, and text generation tasks, long text processing is a common challenge. The present invention effectively reduces the cost of inference by reducing unnecessary tokens involved in calculations. Experiments have shown that compared with the existing Transformer model, this solution can reduce the computational overhead by about 40% while maintaining task performance close to the original model. This optimization scheme is suitable for scenarios with high requirements for computational efficiency, such as large-scale language models and machine translation.

[0039] Compared with the traditional Transformer model, the present invention has the following significant advantages:

[0040] 1. Improved computing efficiency: By dynamically allocating computing budgets, the waste of computing resources is reduced.

[0041] 2. Model performance is maintained: Despite the reduced computational budget, the model still performs well in various natural language processing tasks.

[0042] 3. Flexibility: The calculation path can be flexibly adjusted according to task requirements and the importance of tokens to adapt to different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The transformer block diagram after the router module is introduced into the present invention and the routing is added;

[0044] Figure 2 This is a schematic diagram of the overall operation process of dynamically selecting a token in the present invention. DETAILED DESCRIPTION

[0045] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0046] This paper proposes a hybrid deep model, which optimizes the computational efficiency of the Transformer model by introducing a dynamic computation allocation mechanism. The main technical solutions are as follows:

[0047] The improvement of computing efficiency is mainly achieved through the dynamic computing resource allocation mechanism. This method dynamically weighs the importance of different tokens and flexibly allocates computing budgets, thereby reducing unnecessary computing overhead and maintaining the overall performance of the model. The specific optimization process is as follows:

[0048] Step 1: Dynamic calculation budget setting: In the traditional Transformer model, each token will be processed with the same amount of calculation, ignoring the actual importance of tokens in the sequence. This invention proposes a method for setting a dynamic calculation budget, and the specific steps are as follows:

[0049] Step 1.1: Each Transformer block sets a computational budget threshold B, which indicates the total number of tokens allowed to participate in the self-attention mechanism and multi-layer perceptron (MLP) calculations at the current stage. This threshold is an adjustable hyperparameter and is set according to task requirements.

[0050] Step 1.2: In the initial stage, the model calculates the initial importance weight of each token, sorts the tokens according to the weight value, selects the tokens with higher weights to enter the calculation path, and the remaining low-weight tokens will temporarily skip the calculation of the current block and enter the residual path.

[0051] Step 2: Introduction of the router module: The router module is the core component of the optimization solution, responsible for dynamically allocating computing resources in each Transformer computing block. The specific workflow is as follows:

[0052] Step 2.1. In each Transformer block, the dynamic importance weight w of each token is calculated through the router module. t , w t The weight calculation is done through linear projection and feedforward network. The formula is as follows:

[0053] w t =W·t i +b

[0054] Among them, t i is the embedding representation of the token, W is the weight matrix of the linear projection, and b is the bias term.

[0055] Step 2.3: Based on the calculated weights, use the Top-K mechanism to select the first K tokens with the highest weights to enter the self-attention calculation path, and the remaining tokens will enter the residual path to avoid unnecessary calculation overhead. The specific Top-K selection mechanism is as follows:

[0056] K=TopK(w t ,k)

[0057] Among them, TopK means selecting the first l elements with the highest weight value, k is the user-defined capacity value, which indicates the maximum number of tokens processed in each calculation block, and w t is the importance weight.

[0058] Step 3: Hybrid depth calculation path design: Dynamically allocating calculation paths based on the importance of tokens is another major innovation of the present invention. The specific steps are as follows:

[0059] Step 3.1: After passing through the router module, each token enters a different calculation path. Tokens with higher importance will participate in the calculation of the complete self-attention mechanism and MLP layer. The path is as follows:

[0060] f(t i )=SelfAttention(t i )+MLP(t i )

[0061] Among them, f(t i ) represents the calculation path function, SelfAttention represents the self-attention mechanism operation, and MLP represents the multi-layer perceptron operation.

[0062] Step 3.2: Tokens with lower importance will bypass these calculations and go directly to the next layer through the residual path. The formula is as follows:

[0063] r(t i)=Residual(t i )

[0064] Among them, r(t i ) represents the residual path function, and Residual represents the residual connection operation.

[0065] Step 3.3: At the final output, the tokens of the residual path and the tokens of the calculation path will be merged to ensure that the model can retain the global information of all tokens but reduce unnecessary calculations.

[0066] Step 4: Dynamic calculation adjustment mechanism: During the model training and reasoning process, dynamically adjust the calculation budget and routing strategy according to the calculation requirements at different stages. By gradually increasing the value of Top-K, only a small number of important tokens are involved in the calculation in the early stage, and as the training progresses, more tokens will be gradually introduced into the calculation path. This dynamic adjustment mechanism ensures that the model can maximize the calculation efficiency without losing performance.

[0067] Step 5: Practical application and performance improvement: Experiments have shown that the Transformer model based on this technical solution shows significant improvement in computational efficiency when processing long sequence tasks. For example, in natural language processing tasks, this solution can reduce computational overhead by about 40% while maintaining task performance close to that of the original model. This optimization solution is suitable for scenarios with high computational efficiency requirements, such as large-scale language models and machine translation.

[0068] The specific embodiments are as follows:

[0069] like Figure 1 As shown in the figure, the structural flow of the hybrid depth-based Transformer computing block in the present invention is shown. The process uses a dynamic routing mechanism to optimize computing efficiency and includes a multi-layer self-attention mechanism and a multi-layer perceptron (MLP) structure. The specific workflow is as follows:

[0070] Step (1), Router module: The Router module at the bottom of the figure receives the input Xi and dynamically decides whether to allow the token to enter the calculation path based on the weight value www of each token (as shown in the figure, the weight values ​​are 0.65 and 0.41 respectively). Tokens with larger weights will enter the full calculation path, while tokens with smaller weights will bypass certain calculation steps directly through the residual path.

[0071] Step (2), Self-attention mechanism: If the weight www of Xi exceeds the set threshold, it will be calculated through the Self-attention layer. This calculation layer will generate a new representation based on the relationship between all tokens.

[0072] Step (3), MLP layer: The output of the self-attention layer is normalized and then enters the MLP layer (multi-layer perceptron) for further nonlinear transformation and feature extraction.

[0073] Step (4), residual connection: During the self-attention and MLP calculation process, the residual connection directly adds the original input and output through the addition operation to maintain the continuity of the information flow and avoid information loss.

[0074] Step (5), output: The final output includes the results of the self-attention mechanism and MLP processing, combined with the token information that has not been calculated in the residual path.

[0075] This process effectively reduces the computational burden of unimportant tokens and optimizes the allocation of computing resources, thereby improving the overall efficiency of the model.

[0076] like Figure 2 As shown in the figure, this figure shows how multiple computational blocks in the Transformer model dynamically select different computational paths based on the importance of the token. The figure is divided into two parts, mainly showing how each token in an example sentence "i like llm" enters the model and is assigned to its own computational path in different layers.

[0077] Left part: This is the initial input layer of the Transformer model, which contains three tokens in the sentence: "i", "like", and "llm". These tokens, as input, will enter the Transformer's calculation path and be processed in different layers according to their respective importance.

[0078] The right part shows that during the calculation process, each token is dynamically selected to enter a different TransformerBlock layer. Specifically, different tokens will enter different calculation layers for complete processing, but they will not enter all layers for processing, thereby reducing unnecessary calculation overhead.

[0079] From this diagram, we can clearly see how the model dynamically allocates computing budgets based on different tokens. Different tokens will be processed in different computing layers, but not all computing layers will be used, thus reducing computing resource consumption.

[0080] The core technical point of the present invention is:

[0081] 1. Dynamic computing budget allocation: By dynamically adjusting the computing budget, optimal utilization of computing resources is achieved.

[0082] 2. Design and implementation of router module: The key technical point of the present invention is to dynamically allocate computing resources according to the importance of tokens.

[0083] 3. Selection of calculation path: By accurately selecting the tokens involved in the calculation, the calculation overhead is significantly reduced while maintaining the performance of the model.

[0084] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A transformer calculation optimization method based on hybrid depth, characterized in that: By introducing a dynamic computing allocation mechanism, the computing efficiency of the Transformer model is optimized. The computing efficiency is improved mainly through the dynamic computing resource allocation mechanism. This method dynamically weighs the importance of different tokens and flexibly allocates computing budgets, thereby reducing unnecessary computing overhead and maintaining the overall performance of the model. The specific optimization process is as follows: Step 1: Dynamic calculation budget setting: A dynamic calculation budget setting method, each Transformer block sets a calculation budget threshold B, which indicates the total number of tokens allowed to participate in the self-attention mechanism and multi-layer perceptron calculation at the current stage; Step 2: Introduction of the router module: The router module is the core component of the optimization solution, responsible for dynamically allocating computing resources in each Transformer computing block; The dynamic importance weight of each token is calculated through the router module, and the Top-K mechanism is used to select tokens with high weights to enter the self-attention calculation path. The remaining tokens will enter the residual path; Step 3: Hybrid depth calculation path design: dynamically allocate calculation paths based on the importance of tokens; Tokens with higher importance will participate in the full self-attention mechanism and MLP layer calculations, while tokens with lower importance will bypass these calculations and go directly to the next layer through the residual path; the two are merged at the final output; Step 4: Dynamic calculation adjustment mechanism: During the model training and reasoning process, the calculation budget and routing strategy are dynamically adjusted according to the calculation requirements at different stages. By gradually increasing the value of Top-K, only a small number of important tokens are involved in the calculation in the early stage, and as the training progresses, more tokens will be gradually introduced into the calculation path. Step 5: Application: Apply the Transformer model to process long sequence tasks.

2. The transformer calculation optimization method based on hybrid depth according to claim 1, characterized in that: The specific steps are as follows: Step 1: Dynamically calculate budget settings. The specific steps are as follows: Step 1.

1. Each Transformer block sets a computational budget threshold B, which indicates the total number of tokens allowed to participate in the self-attention mechanism and multi-layer perceptron calculations at the current stage; this threshold is an adjustable hyperparameter and is set according to task requirements; Step 1.2: In the initial stage, the model calculates the initial importance weight of each token, sorts the tokens according to the weight value, selects the tokens with higher weights to enter the calculation path, and the remaining low-weight tokens will temporarily skip the calculation of the current block and enter the residual path; Step 2: Introduction of the router module. The specific workflow is as follows: Step 2.

1. In each Transformer block, the dynamic importance weight w of each token is calculated through the router module. t , w t The weight calculation is done through linear projection and feedforward network; the formula is as follows: w t =W·t i +b Among them, t i is the embedding representation of token, W is the weight matrix of linear projection, and b is the bias term; Step 2.3: Based on the calculated weights, the Top-K mechanism is used to select the first K tokens with the highest weights to enter the self-attention calculation path. The remaining tokens will enter the residual path to avoid unnecessary calculation overhead. The specific Top-K selection mechanism is as follows: K=TopK(in t ,k) Among them, TopK means selecting the first k elements with the highest weight value, k is the user-defined capacity value, which indicates the maximum number of tokens processed in each calculation block, and w t is the importance weight; Step 3: Design of hybrid depth calculation path. The specific steps are as follows: Step 3.1: After passing through the router module, each token enters a different calculation path; tokens with higher importance will participate in the calculation of the complete self-attention mechanism and MLP layer. The path is as follows: f(t i )=SelfAttention(t i )+MLP(t i ) Among them, f(t i ) represents the calculation path function, SelfAttention represents the self-attention mechanism operation, and MLP represents the multi-layer perceptron operation; Step 3.2: Tokens with lower importance will bypass these calculations and go directly to the next layer through the residual path; the formula is as follows: r(t i )=Residual(t i ) Among them, r(t i ) represents the residual path function, and Residual represents the residual connection operation; Step 3.3: At the final output, the tokens of the residual path and the tokens of the calculation path will be merged to ensure that the model can retain the global information of all tokens, but reduce unnecessary calculations; Step 4: Dynamic calculation adjustment mechanism: During the model training and reasoning process, the calculation budget and routing strategy are dynamically adjusted according to the calculation requirements at different stages. By gradually increasing the Top-K value, only a small number of important tokens are involved in the calculation in the early stage, and as the training progresses, more tokens will be gradually introduced into the calculation path. This dynamic adjustment mechanism ensures that the model can maximize the calculation efficiency without losing performance. Step 5: Practical application and performance improvement: Applying the Transformer model to process long sequence tasks can improve computational efficiency.

Images