Long sequence adaptive segmentation method based on front and back request perception
By constructing a prediction model for decoding layer execution time and segmentation loss, and dynamically adjusting the adaptive segmentation of long sequences, the problem of uneven sub-block execution time in pipeline inference is solved, achieving more efficient end-to-end throughput and resource utilization.
Patent Information
- Application Number
- CN202511080273.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-04
- Publication Date
- 2025-11-18
AI Technical Summary
When processing long sequences, existing technologies suffer from uneven execution times of sub-blocks in pipelined inference, resulting in significant uneven bubble times. This affects the efficiency of distributed inference, especially in ultra-long sequences and high-concurrency scenarios where bubble times account for more than 30%, becoming a bottleneck restricting efficiency.
A long-sequence adaptive segmentation method based on prior and subsequent request awareness is adopted. By constructing a decoding layer execution time prediction model and a segmentation loss prediction model, the execution time of the preceding request and the segmentation loss of the current request are dynamically perceived to achieve fine-grained adaptive segmentation, optimize the computation and memory read/write load, and select the segmentation strategy with the shortest total execution time.
It significantly reduces pipeline bubble time by 25%, increases end-to-end throughput by 18%, achieves a balance between computing and read/write loads, and improves hardware resource utilization.
Smart Images
Figure CN120973494A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of neural networks, in particular to a long sequence adaptive segmentation method based on front and rear request perception, which is used to realize the calculation and read-write load balancing in large model pipeline inference and improve the utilization rate of hardware resources. BACKGROUND
[0002] Based on the breakthrough application requirements of current large language models (LLM) in natural language processing, code generation and other fields, the parameter quantity has broken through 100 billion or even 1000 billion (such as GPT-4, DeepSeek-R1, etc.), far exceeding the memory and computing power limit of single node GPU, prompting distributed inference to become an inevitable trend; among them, pipeline inference as an important distributed inference method, by cutting the model into multiple devices according to the layer, and using Micro-Batch scheduling to realize the calculation overlap between stages, effectively solving the single node resource bottleneck problem, significantly improving the throughput and response efficiency.
[0003] The current mainstream pipeline inference optimization scheme such as DeepSpeed-FastGen's Dynamic SplitFuse and vLLM's Chunk-prefill mainly reorganizes the prefill and decode tasks to perform forward calculation with a fixed number of tokens to achieve calculation load balancing and improve device utilization. However, the method of inference with a fixed number of tokens can achieve balanced calculation load, but when processing long sequences, the reading of different lengths of KV Cache by each sub-block will cause differences in memory access time, making the actual execution time still unbalanced, and thus unable to eliminate the pipeline bubble problem. Specifically, when processing long sequences, it needs to be segmented into multiple sub-blocks, and the later sub-blocks need to access the linearly growing KV cache and perform large-scale matrix multiplication, so the total execution time can be significantly longer than the front sub-blocks (several times difference), and thus form a new bubble inside the long sequence. This eventually leads to a significant imbalance in the actual execution time of each stage of the pipeline, causing a cascade of bubble time. Even if the number of tokens is equal each time, in the case of long sequences and high concurrency, the bubble time ratio is still more than 30%, becoming the core bottleneck restricting the efficiency of distributed inference. SUMMARY
[0004] In view of the above defects of the prior art, the present application provides a long sequence adaptive segmentation method based on front and rear request perception, which realizes fine-grained adaptive segmentation of long sequences by dynamically perceiving the front request execution time and the current request segmentation loss, significantly compresses the pipeline bubble time, models and adaptively segments the sub-block calculation time difference based on the sub-block calculation time difference, and reduces the pipeline bubble time by up to 25%, effectively alleviating the cascade waiting problem; at the same time, the end-to-end inference throughput is comprehensively optimized, the device calculation and memory read-write load are dynamically aligned, and the overall end-to-end throughput is improved by up to 18%.
[0005] To achieve the above object, the technical scheme adopted by the present application is as follows:
[0006] A long sequence adaptive segmentation method based on front and rear request perception, comprising the following steps:
[0007] Step 1, constructing a decoding layer execution time prediction model;
[0008] Step 2, collecting decoding layer calculation time fitting data; fitting the dependent variable and the to-be-fitted parameter of the decoding layer execution time prediction model according to the decoding layer calculation time fitting data;
[0009] Step 3, constructing a pipeline bubble time model; the pipeline bubble time includes an initial waiting time and a sub-block calculation time difference;
[0010] Step 4, constructing a segmentation loss prediction model, comprising:
[0011] Step 4.1, quantifying the loss of the actual block number being less than the optimal block number by attention calculation loss;
[0012] Step 4.2, quantifying the loss of the actual block number exceeding the optimal block number by fine-grained segmentation loss;
[0013] Step 5, collecting segmentation loss data; fitting the dependent variable and the to-be-fitted parameter of the segmentation loss prediction model according to the segmentation loss data;
[0014] Step 6, inputting the sequence length and the block number, adaptively segmenting the long sequence according to the decoding layer execution time prediction model to obtain the segmentation strategy and the decoding layer calculation time; calculating the pipeline bubble time according to the pipeline bubble time model; calculating the segmentation loss time according to the segmentation loss prediction model; summing up the decoding layer calculation time, the pipeline bubble time and the segmentation loss time to obtain the total execution time;
[0015] Step 7, traverse all the number of chunks, loop to perform the long sequence adaptive segmentation of step 6, obtain the corresponding segmentation strategy and total execution time; select the segmentation strategy with the shortest total execution time as the best strategy.
[0016] As preferred, in the decoding layer execution time prediction model, the operators affecting the decoding layer execution time include:
[0017] Operator 1, an operator only related to the current sub-block length;
[0018] Operator 2, an operator only related to the historical KVCache length;
[0019] Operator 3, an operator dependent on both the current sequence length and the total Cache length;
[0020] Operator 4, a fixed overhead;
[0021] Merge the to-be-fitted parameters in the operator 1 as x1; let the to-be-fitted parameters in the operator 2 as x2; merge the to-be-fitted parameters in the operator 3 as x3; let the to-be-fitted parameters in the operator 4 as x0; finally obtain the decoding layer computation time prediction model as:
[0022] T compute =x0+x1·chunk_size+x2·len_kv+x3·chunk_size·len_kv.
[0023] As preferred, in step 3, the long sequence has n sub-blocks, and the sub-block computation times are respectively: C0, C1...C n-1 , the pipeline parallelism is m, and the pipeline stages are respectively 0, 1,...m-1;
[0024] The initial waiting time of the pipeline stage j is j times of the computation time C0 of the 0th sub-block;
[0025] When the pipeline stage j processes the subsequent sub-block and the current sub-block execution time is longer than the previous sub-block, the pipeline bubble time is equal to j times of the difference value between the adjacent sub-block computation times; the pipeline bubble time calculation formula is as follows:
[0026]
[0027] Wherein, j is the serial number of the current pipeline stage, C init is the.
[0028] As preferred, C init =C0 the initial waiting time only exists in two scenarios:
[0029] Pipeline start: system processes the first request, needs to build the calculation flow from zero;
[0030] First request after idling: pipeline is reactivated after idling, needs to refill the calculation stage.
[0031] As preferred, the step 4.1 comprises:
[0032] The total attention calculation complexity is calculated by accumulating the calculation amount of each sub-block;
[0033] The length of each sub-block is obtained by the sequence length L and the number of sub-blocks N
[0034] The optimal length L is obtained by the sequence length L and the optimal number of sub-blocks N opt The optimal length L corresponding to the hardware throughput peak is obtained
[0035] According to the total attention calculation complexity, the length of each sub-block C and the optimal length L opt , the attention calculation loss of the number of sub-blocks N compared with the optimal number of sub-blocks N opt is:
[0036]
[0037] Wherein, a is a linear relationship.
[0038] As preferred, in the step 4.2, the fine-grained segmentation loss comprises:
[0039] (1) Additional scheduling loss:
[0040] Additional scheduling loss = b·(N-N opt )
[0041] Wherein, the additional scheduling loss and the difference between the number of sub-blocks N and the optimal number of sub-blocks N opt are linearly related, and the linear relationship is b;
[0042] (2) The fixed overhead proportional to the sequence length is c·L;
[0043] (3) The fixed cost of hardware initialization and framework scheduling is d;
[0044] The fine-grained segmentation loss prediction model is:
[0045] Fine-grained segmentation loss prediction model = b·(N-N opt )+c·L+d.
[0046] As preferred, the segmentation loss prediction model is:
[0047]
[0048] As preferred, the segmentation loss data comprises total execution time data under different sequence length and segmentation number combinations.
[0049] As preferred, both the step 2 and the step 5 adopt linear least square method for fitting.
[0050] Compared with the prior art, the beneficial effects of the present application are embodied in:
[0051] 1. Based on real-time hardware load, dynamic calculation of differentiated sub-block size (instead of fixed Token number), through sub-block calculation time prediction model, the calculation time of each sub-block is ensured to be close to consistent, so as to compress the internal bubble time of the request to the minimum; at the same time, through traversing different block numbers to generate multiple groups of candidate strategies, a decision basis is provided for global optimization.
[0052] 2. The actual execution time of the predecessor request is perceived, the bubble time of all segmentation strategies of the current request is predicted, the segmentation strategy that can be aligned with the execution time of the predecessor request is selected, so as to reduce the inter-request bubble; the segmentation loss time is predicted, and the segmentation loss is controlled within a certain range. Through the inter-request bubble time and the request segmentation loss control, the maximum global throughput is realized.
[0053] The scheme dynamically takes into account the single request segmentation loss (communication overhead, scheduling delay), the cross-request and inter-request bubble accumulation effect, ensures that the selected sub-block size not only adapts to the real-time hardware load, but also can be aligned with the calculation time of the predecessor request, effectively suppresses the pipeline bubble time. Finally, the end-to-end bubble is minimized, and in the scene where the request length difference is large, the throughput and resource utilization can still be significantly improved. BRIEF DESCRIPTION OF DRAWINGS
[0054] Figure 1 It is a pipeline bubble time calculation schematic diagram of the embodiment 1 of the present application.
[0055] Figure 2 It is a total attention complexity schematic diagram of the embodiment 1 of the present application.
[0056] Figure 3 It is a long sequence adaptive segmentation strategy acquisition flowchart of the embodiment 1 of the present application. DETAILED DESCRIPTION
[0057] In order to make the technical means, creative features, purposes and effects of the application easy to understand, the present application will be further described in conjunction with specific drawings. However, the present application is not limited to the following embodiments.
[0058] It is to be understood that the structures, proportions, sizes, etc. shown in the drawings accompanying the present specification are merely used to cooperate with the content disclosed in the present specification for understanding and reading by those skilled in the art, and are not used to limit the defined conditions under which the present application can be implemented, and therefore do not have technical substantive significance. Any modification of the structure, change of the proportional relationship or adjustment of the size, without affecting the effects that can be produced by the present application and the purposes that can be achieved, should still fall within the scope of the technical content disclosed by the present application.
[0059] Embodiment 1:
[0060] A long sequence adaptive segmentation method based on front and rear request perception, comprising the following steps:
[0061] Step 1, constructing a decoding layer calculation time prediction model;
[0062] The decoding layer (DecoderLayer) is the core unit of the autoregressive inference of the large language model. Based on the decoding layer calculation process, the operators that affect the decoding layer calculation time are divided into four categories:
[0063] Operator 1: Operator only linearly related to chunk size.
[0064] It includes linear transformation (Linear) such as Query vector generation, feedforward network, Softmax activation function, normalization operation such as RMSNorm / LayerNorm, and KVCache write operation. The calculation time of this type of operator is linearly and positively related to the chunk size, and is irrelevant to the history context. The modeling formula of this part of the operator is: k i *chunk_size, the calculation process involved is shown in Table 1:
[0065] Table 1 Modeling formula of operator linearly related to chunk size
[0066] Computational process Modeling equation Attention Norm k1 * chunk_size QK VO matrix computation k2 * chunk_size KV write Cache k3 * chunk_size Softmax k4 * chunk_size FFN Norm k5 * chunk_size FFN k6 * chunk_size
[0067] Operator 2: Operator only related to KVCache total length (len_kv). It is mainly the operation of reading KVCache in the attention mechanism. This operation needs to load all historical caches (including the newly added KV of the current chunk), and its overhead is proportional to the number of tokens accumulated in history.
[0068] It covers the reading, splicing and cache traversal operation limited by the memory bandwidth of the historical KVCache. The time consumption of this type of operator is linearly and positively related to the total length of KVCache. Before self-attention calculation, the historical cache needs to be loaded from the video memory to the calculation unit, and its time consumption increases linearly with the increase of the cache length. The modeling formula of this part of the operator is: k7*len_kv.
[0069] Operator 3: Operator that depends on both sub-block length and total KVCache length:
[0070] Attention score calculation: Matrix multiplication between Query vector and Key vector, with computational complexity of O(chunk_size x len_kv).
[0071] Attention score weighted summation: Multiplication between attention probability matrix and Value vector, with computational complexity of O(chunk_size x len_kv).
[0072] In the operator involving the joint action of sub-block length and total KVCache length, the time-consuming of its calculation is proportional to the product of sub-block length and total KVCache length. The modeling formula of this part of the operator is: k l chunk_size x len_kv, and the calculation process involved is shown in Table 2:
[0073] Table 2 Modeling formula of operator with linear relationship with the product of sub-block size and total KVCache length
[0074] Computational process Modeling equation S = QK t ]] k8 * chunk_size * len_kv SV k9 * chunk_size * len_kv
[0075] Operator 4: Fixed overhead: Constant level fixed overhead. This type of operator involves fixed operations such as initialization of computation graph, pre-allocation of memory, waiting for cross-device synchronization, etc., which are independent of sequence length. The time consumption of such operators is determined by hardware architecture and framework scheduling strategy, such as hardware initialization, computation graph scheduling, memory allocation, etc. The fixed cost. This part of the time consumption in the model is reflected as a constant term, which is independent of the sub-block size and the cache length, reflecting the inherent overhead of hardware and framework. The modeling formula of this part is: k0.
[0076] Combine the to-be-fitted parameters k0-k6 in Operator 1 into x1; let the to-be-fitted parameter k7 in Operator 2 be x2; combine the to-be-fitted parameters k8, k9 in Operator 3 into x3; let the to-be-fitted parameter k0 in Operator 4 be x0; finally obtain the decoding layer computation time prediction model as:
[0077] T compute = x0 + x1 · chunk_size + x2 · len_kv + x3 · chunk_size · len_kv
[0078] Step 2, sampling Set decoding layer computation time fitting data; according to decoding layer computation time fitting data Fit the dependent variable and to-be-fitted parameters of the decoding layer computation time prediction model;
[0079] Sample the decoding layer computation time fitting data under different computation and read-write loads. To accurately quantify the computation time of the decoding layer under different computation and read-write loads, a discretization parameter sampling strategy is used. The sampling scheme is as follows:
[0080] Current sub-block length: Set 1k, 2k, 3k, 4k (unit: token) four typical values, covering the load range from short text generation to long document continuation.
[0081] History KV Cache length sampling: with 1k as a step, from 1k to 16k (unit: token), quantify the relevance of cache read delay and attention calculation.
[0082] By controlling the combined changes of current sub-block length and history KV Cache total length, the sampling can reflect the actual inference load dataset, and provide decoding layer computation time fitting data under different computation and read-write load for decoding layer execution time prediction model fitting.
[0083] Fit the decoding layer execution time prediction model. The dependent variable in the decoding layer execution time prediction model is T compute , the parameter x0 to be fitted represents the constant level fixed overhead when executing sequence computation, x1 represents the linear relationship between linear operator computation time and sub-block length, x2 represents the time of reading KVCache and the linear relationship between KVCache total length, and x3 represents the attention operator computation time and the product of sub-block length and KVCache total length. The linear characteristics of such parameters make the partial derivative of the objective function (execution time T compute ) with respect to each parameter a constant or a linear function, so that the normal equations can be constructed by directly taking the derivative and setting the derivative to zero, and the optimal parameter values x0-x3 can be efficiently solved.
[0084] Step 3, construct the bubble time calculation formula;
[0085] Calculate the bubble time in pipeline parallelism. Assume that there are n sub-blocks (0, 1,...n-1) in total, and the computation time of sub-block i is C i (i=0, 1, 2,..., n-1), and the pipeline parallelism is m, as shown in Figure 1 , p0-p3 represent the four pipeline stages; C1-C4 represent the computation times of the four sub-blocks; B1-B3 represent the bubble time caused by the difference in execution time of sub-blocks C2 and C3 in different pipeline stages. The pipeline bubble mainly has two parts:
[0086] (1) Initial waiting time: pipeline stage j needs to wait for all previous stages to complete the first sub-block computation before it can start computation, so the initial waiting time of pipeline stage j is j times the computation time of the first sub-block, as shown in Figure 1 , the initial waiting time of pipeline stage p3 is 3*C0. Note that if there is an initial waiting time, C init =C0, the initial waiting time only occurs in two scenarios:
[0087] 1. Pipeline start: the system processes the first request, which needs to build the computation flow from zero.
[0088] 2. First request after idling: the pipeline is reactivated after idling, which needs to refill the computation stage.
[0089] If the above two scenarios are not met, there is no initial waiting time, C init = 0.
[0090] (2) Sub-block computation time difference: the execution time of adjacent sub-blocks is different, which will cause pipeline bubbles, and the size of the bubble is equal to the j times of the difference between the execution time of adjacent sub-blocks, where j is the sequence number of the current pipeline stage. Taking C2 and C3 sub-blocks as an example: the computation time of C3 is longer than that of C2, which causes a bubble in pipeline stage 1, and the time of the bubble is (C3-C2); when the bubble is transmitted to stage 2, the bubble time becomes 2*(C3-C2); when it is transmitted to stage 3, it becomes 3*(C3-C2). Conversely, when the computation time of the subsequent sub-block is less than or equal to that of the previous sub-block (such as the computation time of C4 being less than that of C3), no bubble will be generated between the sub-blocks. Therefore, the bubble time of pipeline stage j is calculated as follows:
[0091]
[0092] Step 4, build a segmentation loss prediction model;
[0093] Modeling the long sequence segmentation loss prediction model. The long sequence segmentation loss is the loss of the number of blocks deviating from the optimal number. Too many or too few blocks will cause the overall performance to decline. By quantifying the impact of different block numbers on system performance, a correlation model between block number and total cost is built to provide decision basis for dynamically selecting the optimal segmentation strategy.
[0094] Sub-step 4.1, build attention calculation loss caused by too few blocks;
[0095] When the actual sequence length exceeds the optimal length (L opt ) corresponding to the hardware throughput peak, single-block calculation will cause the attention calculation complexity to increase sharply due to exceeding the hardware parallelism capability; and the deviation of the actual block number N from the optimal block number N opt directly determines the adaptation efficiency of attention calculation and hardware throughput characteristics.
[0096] As shown in Figure 2 , the total attention calculation complexity is composed of the calculation amount of each sub-block (black part), where L is the sequence length, N is the block number, and the length of each sub-block is L / N. The total attention calculation complexity is:
[0097]
[0098] The number of blocks N is calculated based on the formula for the total attention complexity, compared to the optimal number of blocks N. opt The attention calculation loss is:
[0099]
[0100] From the sequence length L and the number of blocks N, the length of each block can be obtained as follows: Given the sequence length L and the optimal number of blocks N opt The optimal length corresponding to the peak hardware throughput can be obtained as follows: C and L opt Substituting into the above formula and simplifying, we get:
[0101]
[0102] The formula shows that the attention calculation loss is linearly related to the block length C. Let the linear relationship be denoted as 'a', and the final attention calculation loss prediction model can be:
[0103]
[0104] Sub-step 4.2: Construct a fine-grained segmentation loss prediction model for excessive number of blocks;
[0105] The number of blocks introduces additional linear overhead, including increased cross-device communication frequency, initialization overhead from repeated computation graph calls, and scheduling latency due to frequent memory lock contention. These overheads are approximately positively correlated with the number of blocks, necessitating a direct constraint on the upper limit of the block count via linear terms to prevent resource contention and performance degradation caused by excessively fine-grained partitioning. The fine-grained partitioning loss prediction model consists of the following parts:
[0106] 1. Additional scheduling overhead. Additional scheduling overhead is proportional to the deviation from the theoretically optimal number of blocks, mainly including cross-device communication time, context management overhead for task switching, and the cost of repeated sub-block reads. When the actual number of blocks is greater than the optimal value, the linear overhead caused by fine-grained partitioning increases significantly; conversely, when the number of blocks is less than the optimal value, this type of overhead decreases accordingly. The overhead caused by additional scheduling times and the number of blocks N compared to the optimal number of blocks N. opt The difference exhibits a linear relationship; let the linear relationship be b. The loss prediction model corresponding to the additional scheduling loss is as follows:
[0107] Additional scheduling loss prediction model = b·(NN) opt )
[0108] 2. Fixed overhead proportional to sequence length, such as activation value storage and memory pre-allocation; this part of the prediction model is c·L
[0109] 3. The fixed cost d of hardware initialization and framework scheduling (e.g. computation graph compilation, cache warm-up).
[0110] The fine-grained split loss prediction model is:
[0111] The fine-grained split loss prediction model = b·(N-N opt )+c·L+d
[0112] The split loss prediction model is obtained by combining the attention calculation loss prediction model and the fine-grained split loss prediction model:
[0113]
[0114] T loss (L, N, C) is defined as: for a certain sequence length, its value is equal to the difference between the total execution time corresponding to a certain number of blocks and the total execution time corresponding to the optimal number of blocks, which quantifies the loss of the current number of blocks relative to the theoretical optimal number.
[0115] This constraint method introduces a dual regulation mechanism of fine-grained split loss and attention calculation loss to ensure that the number of sub-blocks N is within a reasonable range: when the number of sub-blocks N is too large, the fine-grained loss increases significantly, effectively inhibiting excessive splitting; when the number of sub-blocks N is too small, the attention calculation loss rises sharply, effectively constraining the case of insufficient splitting.
[0116] Step 5, collect split loss data; fit the dependent variable and the to-be-fitted parameters of the split loss prediction model according to the split loss data.
[0117] Split loss data sampling: based on the long sequence split loss prediction model, the system samples the total execution time data under different combinations of sequence length and block number, and the specific scheme is as follows:
[0118] Select five typical values of sequence length as 8k, 10k, 12k, 14k, and 16k (unit: token), covering the scenarios from medium-length text generation to ultra-long document processing; at the same time, set the number of sub-blocks from 2 to 16 in increments, quantifying the impact of the number of sub-blocks on computational efficiency.
[0119] Long sequence split loss prediction model fitting: based on the structural characteristics of the loss function, the target variable T loss is in a linear relationship with the first-order term of the to-be-fitted parameters (a, b, c, d). The to-be-fitted parameters a, b, c, and d respectively represent the linear relationship between the loss of different parts and the independent variable. Therefore, the linear least squares method can directly minimize the sum of squares of residuals between the predicted value and the actual loss, achieving high-precision fitting of the parameters.
[0120] Step 6, long sequence adaptive cut is performed and a cut strategy is returned. The length of the sequence and the number of blocks are input, and based on the above decoding layer calculation time prediction model, the bubble time calculation formula and the cut loss time prediction model, the decoding layer calculation time, the bubble time and the cut loss are calculated. The sum of the above three times is the total execution time, and finally the cut strategy and the total execution time are returned. The detailed steps are as follows:
[0121] Step 6.1, cut strategy and total calculation time acquisition. The flow is as shown in Figure 3 , and the calculation steps are as follows:
[0122] 1. Input the request length and the number of blocks, and based on the fixed token cut strategy, the execution time of each sub-block is calculated and the average T is obtained.
[0123] 2. The average T obtained is used as the dependent variable T compute of the calculation time prediction model, and the current sub-block length chunk_size is solved based on the decoding layer execution time prediction model and the historical KVcache total length len_kv is accumulated.
[0124] 3. Repeat step 2 until the cut is completed, and return the solved cut strategy and the execution time of each sub-block.
[0125] Step 6.2, total bubble time acquisition: based on the bubble time calculation formula, the total bubble time is obtained.
[0126] Step 6.3, total cut loss time acquisition: input the request length and the number of blocks, and based on the long sequence cut loss prediction model, the cut loss time is obtained.
[0127] Step 7, best cut strategy selection. Based on the sequence length L, all feasible block numbers are traversed, and the long sequence adaptive cut process of step 6 is called in a loop to obtain the key indicators under each cut strategy:
[0128] Decoding layer calculation time: the average time consumption of each sub-block is dynamically calculated through the decoding layer time prediction model, the chunk_size is deduced in reverse and the len_kv is accumulated, and the cut strategy and the decoding layer calculation time are returned after the calculation is completed;
[0129] Bubble time: the total bubble time is quantified by combining the inter-request bubble (the time difference between the last sub-block of the previous request and the first sub-block of the current request) and the intra-request bubble (the execution time difference between adjacent sub-blocks);
[0130] Cut loss prediction model: based on the cut loss prediction model, the loss value of the current cut number is output.
[0131] The total execution time is obtained by summing up the three parts of time, and the system records the total execution time corresponding to each cutting frequency and its cutting strategy. Finally, the strategy with the minimum total execution time is selected as the best cutting strategy.
[0132] The core innovation of the long sequence adaptive cutting strategy is to systematically solve the three key problems of load imbalance, low scheduling efficiency, and large pipeline bubble proportion in traditional cutting methods. Specifically, it embodies in:
[0133] Dynamic load balancing mechanism: Through the inverse solution of the chunk size by the decoding layer execution time prediction model, combined with the historical KV length (len_kv) accumulation calculation, the computing load of each sub-block is dynamically adjusted to ensure that the execution time of all sub-blocks within the request is close to equal, eliminating the performance bottleneck caused by fixed cutting from the source.
[0134] Intelligent scheduling gap optimization: Innovatively quantifies the inter-request bubble time (the time difference between the last sub-block of the predecessor request and the first sub-block of the current request) and the intra-request bubble time (the execution time difference between adjacent sub-blocks), and adjusts the current cutting strategy in real time based on the execution time of the last sub-block of the predecessor request to minimize the inter-request bubble time.
[0135] Cutting loss modeling: Introduces a cutting loss prediction model based on linear least squares method to accurately evaluate the efficiency loss value (the difference between actual time consumption and theoretical minimum time consumption) under different sequence length (L) and block number (N) combinations, effectively suppressing the rapid increase of computing and communication overhead caused by extreme block number, achieving global balance between computing gain and communication loss.
[0136] The synergistic effect of the above innovations enables the system to dynamically select the strategy with the globally minimum total execution time (computing + bubble + loss) after traversing all feasible block numbers, ultimately achieving sub-block load balancing, scheduling real-time performance, and controllability of computing and communication overhead, providing an efficient solution for ultra-long sequence processing.
[0137] This embodiment implements and verifies a long sequence adaptive cutting method based on predecessor request duration and current request cutting loss perception on the vllm-mindspore framework. The Qwen2.5-32B model and Longbench long text dataset are used for testing on the Ascend910B2 computing platform. The experiment shows that this method dynamically generates candidate cutting strategies and integrates predecessor request duration perception mechanism, which improves the prefill stage throughput by up to 25% and the end-to-end throughput by up to 18% compared to vllm-mindspore.
[0138] Based on the above technical scheme and experimental verification, the method finally realizes a substantial breakthrough in the load balancing mechanism in long sequence pipeline reasoning, and provides key support for efficient processing of long text by large models.
Claims
1. A long sequence adaptive segmentation method based on prior and subsequent request awareness, characterized in that, Includes the following steps: Step 1: Construct a decoding layer execution time prediction model; Step 2: Collect decoding layer computation time fitting data; fit the dependent variable and the parameters to be fitted in the decoding layer execution time prediction model based on the decoding layer computation time fitting data; Step 3: Construct a pipeline bubble time model; pipeline bubble time includes initial waiting time and the difference in sub-block computation time; Step 4: Construct a segmentation loss prediction model, including: Step 4.1: Quantify the loss when the actual number of blocks is less than the optimal number of blocks by calculating the loss through attention; Step 4.2: Quantify the loss when the actual number of blocks exceeds the optimal number of blocks using fine-grained segmentation loss; Step 5: Collect segmentation loss data; fit the dependent variable of the segmentation loss prediction model with the parameters to be fitted based on the segmentation loss data; Step 6: Input the sequence length and the number of blocks. Perform adaptive segmentation on the long sequence according to the decoding layer execution time prediction model to obtain the segmentation strategy and decoding layer computation time. Calculate the pipeline bubble time according to the pipeline bubble time model. Calculate the segmentation loss time according to the segmentation loss prediction model. Sum the decoding layer computation time, the pipeline bubble time, and the segmentation loss time to obtain the total execution time. Step 7: Traverse all the number of blocks, repeatedly execute the long sequence adaptive segmentation in Step 6, obtain the corresponding segmentation strategy and the total execution time; select the segmentation strategy with the shortest total execution time as the best strategy.
2. The adaptive segmentation method for long sequences based on prior and subsequent request awareness as described in claim 1, characterized in that, In the decoding layer execution time prediction model, the operators affecting the decoding layer execution time include: Operator 1 is an operator that is only related to the length of the current sub-block; Operator 2, an operator that is only related to the length of the historical KVCache; Operator 3 is an operator that depends on both the current sequence length and the total cache length; Operator 4, fixed cost; The parameters to be fitted in operator 1 are merged into x1; the parameters to be fitted in operator 2 are set to x2; the parameters to be fitted in operator 3 are merged into x3; the parameters to be fitted in operator 4 are set to x0; finally, the decoding layer computation time prediction model is obtained as follows: T compute =x0+x1·chunk_size+x2·len_kv+x3·chunk_size·len_kv。 3. The adaptive segmentation method for long sequences based on prior and subsequent request awareness as described in claim 1, characterized in that, In step 3, the long sequence has a total of n sub-blocks, and the computation time of each sub-block is C0, C1...C... n-1 The pipeline parallelism is m, and the pipeline stages are 0, 1, ... m-1; the initial waiting time of pipeline stage j is j times the computation time C0 of the 0th sub-block. When pipeline stage j processes subsequent sub-blocks, if the execution time of the current sub-block is longer than that of the previous sub-block, the pipeline bubble time is equal to j times the difference in computation time between adjacent sub-blocks; the formula for calculating the pipeline bubble time is as follows: Where j is the sequence number of the current pipeline stage, C init As stated.
4. The adaptive segmentation method for long sequences based on prior and subsequent request awareness according to claim 3, characterized in that, C init =C0 The initial waiting time mentioned only exists in two scenarios: When the pipeline starts: the system processes the first request, and the computation flow needs to be built from scratch; The first request after idle: The pipeline is reactivated after being idle, and the calculation phase needs to be refilled.
5. The adaptive segmentation method for long sequences based on prior and subsequent request awareness according to claim 1, characterized in that, Step 4.1 includes: Calculate the total attention computational complexity, which is the sum of the computational costs of each sub-block. The length of each block is obtained from the sequence length L and the number of blocks N. The sequence length L and the optimal number of blocks N opt The optimal length corresponding to the peak hardware throughput is obtained as follows Based on the total attention computation complexity, the length of each block C, and the optimal length L opt The number of blocks N is obtained compared with the optimal number of blocks N. opt The attention calculation loss is: Where 'a' represents a linear relationship.
6. The adaptive segmentation method for long sequences based on prior and subsequent request awareness according to claim 5, characterized in that, In step 4.2, the fine-grained segmentation loss includes: (1) Additional scheduling overhead: Additional scheduling overhead = b·(NN) opt ) Wherein, the additional scheduling overhead and the number of blocks N are related to the optimal number of blocks N. opt The differences exhibit a linear relationship; let the linear relationship be b. (2) The fixed overhead, which is proportional to the sequence length, is c·L; (3) Fixed cost d for hardware initialization and framework scheduling; The fine-grained segmentation loss is: Fine-grained segmentation loss = b·(NN) opt )+c·L+d.
7. The adaptive segmentation method for long sequences based on prior and subsequent request awareness according to claim 6, characterized in that, The segmentation loss prediction model is as follows:
8. The adaptive segmentation method for long sequences based on prior and subsequent request awareness according to claim 1, characterized in that, The segmentation loss data includes total execution time data under different combinations of sequence length and number of segmentations.
9. The adaptive segmentation method for long sequences based on prior and subsequent request awareness according to claim 1, characterized in that, Both steps 2 and 5 employ linear least squares fitting.