Large model reasoning acceleration method and system based on adaptive speculation decoding

Abstract

The invention provides a large model reasoning acceleration method and system based on adaptive speculation sampling, and belongs to the field of electronic information. According to the method, the throughput is estimated by predicting the number of accepted lexical elements and the running time needed by current iteration, whether speculation decoding is executed or not is determined through a speculation controller, a draft sequence set is updated through a draft model, the updated throughput is estimated till the speculation length reaches the optimal value of the optimal throughput, and the optimal throughput is obtained. High-acceptance-rate lexical elements are screened out through a confidence priority verifier, and an output sequence set of speculation decoding is obtained after verification of a target model. The large model reasoning acceleration method and system based on adaptive speculation sampling can adaptively reduce reasoning delay in various real large model reasoning scenes.

Description

Technical Field

[0001] The present invention belongs to the field of electronic information, and in particular relates to a large model reasoning acceleration method and system based on adaptive speculative decoding. Background Art

[0002] Big models have demonstrated their versatility across a wide range of tasks, including translation, question answering, and mathematical reasoning. This achievement has led to the rapid development of big model services that can perform big model inference and provide responses to user queries in real time. However, in big model inference, autoregressive decoding that sequentially generates output tokens is often the dominant factor in request latency.

[0003] To solve this problem, the field of artificial intelligence began to consider the use of speculative decoding methods. Speculative decoding introduces an efficient drafter (i.e., draft model) to autoregressively generate multiple draft words, which will be later verified in parallel by the original large model (i.e., target model), so that some draft words become output words. The speculative decoding method that generates multiple output words at one time can reduce the inference delay of large models.

[0004] Existing speculative decoding methods have difficulty achieving optimal latency in real large-model inference scenarios due to their preset speculative length. The speculative length controls the number of tokens generated by the draft model and affects latency. Although increasing the speculative length can generate more output tokens, it may also cause resource waste. In real large-model inference scenarios, speculative decoding needs to adapt to changing request patterns and system configurations, and a fixed speculative length is difficult to always match the optimal value. Summary of the invention

[0005] The present invention is made to solve the above-mentioned problems, and its purpose is to provide a large model reasoning acceleration method and system based on adaptive speculative decoding.

[0006] The present invention provides a large model reasoning acceleration method based on adaptive speculative decoding, which has the following characteristics and is used for efficient reasoning services with service level target perception, low end-to-end delay and high resource utilization, including the following steps: S01, setting the historical average confidence of the draft word S02, according to the input sequence set X = {x1, ..., x BS}, set the batch size to BS and the speculation length SL = 0, where x represents the input sequence; S03, calculate the number of predicted accepted tokens NAT: In the formula, p i is the length of the i-th draft sequence, AR i,j represents the probability that the jth draft word in the i-th draft sequence is accepted, c i,k represents the confidence score of the kth draft word in the i-th draft sequence, ci,0 =1, S04, calculate the running time T based on the time model sd , where the running time T sd Including draft model M d The execution time T d and the target model M t The execution time T v ; S05, according to the running time T sd , service level objective constraint tpot and predicted number of accepted tokens NAT, use the service level objective-aware efficiency estimator to calculate the throughput thpt under the current speculation length SL SL ; S06, determine the throughput thpt SL The corresponding speculative length SL, if SL = 0, set SL = 1 and repeat steps S03 to S05 until SL ≠ 0; S07, if SL ≠ 1 or the speculative controller decides to perform speculative decoding, then enter step S08, if SL = 1 or the speculative controller decides not to perform speculative decoding, then input the input sequence set X into the target model M t After forward propagation calculation, the acceptance sequence set Z = {z1, ..., z BS} to update the input sequence set X and enter step S12, where z i =(z i,1 ) is the i-th accepted sequence; S08, the input sequence set X={x1,…,x BS} and the generated draft sequence set Y = {y1,…,y BS} The concatenated draft input sequence set {(x1,y1),…,(x BS ,y BS )} Input draft model M d After the forward propagation calculation, the draft sequence set Y is updated according to the output distribution and the greedy sampling standard, and its confidence is recorded and the historical average confidence of the word unit is calculated. Update, where y i =(y i,1 ,…,y i,SL-1 ) is the i-th draft sequence, i=1,…,BS; S09, calculate the current predicted number of accepted words NAT and the throughput thpt under the current speculation length SL SL After that, according to the throughput thpt SL The speculative length SL is adjusted using the speculative controller and returns to step S03 for iteration until the speculative length SL reaches the optimal value SL * ; S10, use the confidence priority verifier to filter out the currently generated draft sequence set Y = {y1, ..., y BS}, where is the i-th draft sequence; S11, input sequence set X = {x1,…,x BS} and the draft sequence set Y'={y'1,...,y' BS}The concatenated efficient draft input sequence set {(x1,y'1),…,(x BS ,y' BS )} Input target model M t Perform forward propagation calculations, for each draft sequence Generate an acceptance sequence set Z = {z1,…,z BS} and update the input sequence set X, where i = 1, ..., BS, p i is the length of the i-th draft sequence, q i is the accepted word length of the i-th draft sequence in Y'; S12, remove the input sequence with the end mark generated by the large model from the input sequence set X, add the newly received input sequence, if the input sequence set X is not empty, enter step S02 to continue the speculative decoding process, if the input sequence set X is empty, terminate the iteration.

[0007] The large model reasoning acceleration method based on adaptive speculative decoding provided by the present invention may also have the following features: wherein, in step S04: In the formula, α d ,γ d ,δ d For the draft model M d The performance coefficient on a given computing platform, N dc (i) and N db (i) are the draft models M d The number of context tokens and batch tokens during the i-th forward propagation, In the formula, p i is the length of the i-th draft sequence, α v ,γ v ,δ v The target model M t Performance coefficient on a given computing platform.

[0008] The large model inference acceleration method based on adaptive speculative decoding provided by the present invention may also have the following features: wherein, in step S05, if T sd >tpot, then thpt SL =-1; if T sd ≤tpot, then

[0009] In the large model reasoning acceleration method based on adaptive speculative decoding provided by the present invention, it can also have the following characteristics: wherein the forward propagation calculation formula is as follows:

[0010] Where M model represents the model that performs forward propagation, M model ∈{M d ,M t}, i = 1, ..., BS, p i is the length of the i-th draft sequence, the draft model M d and the target model M t have the same vocabulary size.

[0011] The large model inference acceleration method based on adaptive speculative decoding provided by the present invention may also have the following features: wherein, in step S07, the step of updating the input sequence set X is as follows: S07-1, forward propagation calculation is performed on each draft sequence to obtain the probability distribution P(z i,1 |x 1:n ); S07-2, calculate and obtain the acceptance sequence set z: z i =(z i,1 ), S07-3, obtain the updated input sequence set X = {x1, ..., x BS}, x i =(x i ,z i ).

[0012] The large model inference acceleration method based on adaptive speculative decoding provided by the present invention may also have the following features: wherein step S08 includes the following sub-steps: S08-1, the draft sequence set Y is processed by the draft model M d The probability distribution P(y 1,SL |x 1:n ,y 1:SL-1 ),…,P(y BS,SL |x 1:n ,y 1:SL-1 ), use the following calculation formula for greedy sampling: In the formula, y i,SL Indicates the draft model M d The prediction result of the SLth draft word in the i-th draft sequence, i = 1, ..., BS; S08-2, y i,SL Splice to y i After that, y i =(y i,1 ,…,y i,SL-1 ,y i,SL), complete the update of the draft sequence set Y, S08-3, use the following calculation formula to update the historical average confidence c i,SL =max(P(y i,SL |x 1:n ,y 1:SL-1 )), N = N + BS, where N is the total number of historical draft tokens generated by speculative decoding.

[0013] In the large model inference acceleration method based on adaptive speculative decoding provided by the present invention, it can also have the following characteristics: wherein, in step S09, the speculative controller uses the current speculative length to reduce thpt SL With thpt SL-1 Compare and determine whether to continue to increase the speculation length: if thpt SL >thpt SL-1 , it means that the increase of the current speculation length can improve the throughput, and the speculation controller will decide to continue to increase the speculation length and increase the value of SL by 1. If thpt SL ≤thpt SL-1 , it means that the increase of current speculation length cannot improve the throughput, and the speculation length has reached the optimal value. The optimal value SL * =SL-1.

[0014] The large model inference acceleration method based on adaptive speculative decoding provided by the present invention may also have the following features: wherein step S10 includes the following sub-steps: S10-1, the confidence priority verifier generates a draft sequence set Y = {y1, ..., y BS}, check the acceptance rate of each draft word one by one in the order of acceptance rate from low to high, and remove the word with the lowest current acceptance rate from the draft word set; S10-2, calculate the throughput thpt' after removing the word with the lowest current acceptance rate SL , if thpt' SL >thpt SL , then return to step S10-1, if not, then restore the last removed word unit and output the updated draft word unit set Y'={y'1,...,y' BS}, where y' i Represents the i-th draft sequence after elimination.

[0015] In the large model inference acceleration method based on adaptive speculative decoding provided by the present invention, it can also have the following characteristics: wherein, in step S11, the step of updating the input sequence set X is as follows: S11-1, forward propagation calculation is performed on each draft sequence to obtain a probability distribution S11-2, calculate the acceptance sequence set Z: Where i = 1,…,BS, j = 1,…,p i ; S11-3, obtain the updated input sequence set X = {x1,…,x BS}, x i =(x i ,z i ).

[0016] The present invention also provides a large model reasoning acceleration system based on adaptive speculative decoding, which has the following characteristics: using any of the aforementioned large model reasoning acceleration methods based on adaptive speculative decoding, including: an initialization module for allowing a user to set the initial historical average confidence of a draft word And the input sequence set X = {x1,…,x BS}Related parameters; Throughput evaluation module, used to calculate the number of predicted accepted words NAT, running time T sd And the throughput thpt SL ; Dynamic speculation controller, used to dynamically adjust the speculation length SL to the optimal value SL * ; a confidence-priority validator for screening high-acceptance draft words; a draft model inference module, which is controlled by a dynamic speculative controller to decide whether to update the draft sequence set Y; and a target model inference module, which is used to generate a verified output sequence and update the input data.

[0017] Functions and Effects of the Invention

[0018] According to the large model inference acceleration method and system based on adaptive speculative decoding involved in the present invention, the throughput is estimated by predicting the number of accepted words and the running time required for the current iteration, and then the speculative controller decides whether to execute speculative decoding, and the draft sequence set is updated by the draft model and the updated throughput is estimated until the speculation length reaches the optimal value for obtaining the optimal throughput, and then the confidence-first verifier screens out high acceptance rate words, and after verification by the target model, the output sequence set of speculative decoding is obtained.

[0019] Therefore, the large model reasoning acceleration method and system based on adaptive speculative decoding of the present invention can adaptively reduce the reasoning delay in various real large model reasoning scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of a large model reasoning acceleration method based on adaptive speculative decoding in an embodiment of the present invention;

[0021] Figure 2It is an architectural diagram of a large model inference acceleration method based on adaptive speculative decoding in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate a large model inference acceleration method and system based on adaptive speculative decoding of the present invention.

[0023] Figure 1 It is a flowchart of a large model reasoning acceleration method based on adaptive speculative decoding in an embodiment of the present invention; Figure 2 It is an architectural diagram of a large model inference acceleration method based on adaptive speculative decoding in an embodiment of the present invention.

[0024] like Figure 1 and 2 As shown, this embodiment provides a large model reasoning acceleration method based on adaptive speculative decoding for efficient reasoning services with service level objective awareness (SLO), low end-to-end latency, and high resource utilization, including the following steps:

[0025] S01, set the historical average confidence of the draft token

[0026] S02, according to the input sequence set X = {x1, ..., x BS}, set the batch size to BS, the speculation length SL = 0,

[0027] Here, x represents the input sequence.

[0028] S03, based on the batch size BS, the speculation length SL and the historical average confidence of the draft token of the input sequence set X Calculate the number of predicted accepted words NAT:

[0029]

[0030] In the formula, p i is the length of the i-th draft sequence, AR i,j represents the probability that the jth draft word in the i-th draft sequence is accepted.

[0031] Acceptance rate i,j It can be expressed approximately as:

[0032]

[0033] In the formula, c i,k represents the confidence score of the kth draft word in the i-th draft sequence, c i,0 =1,

[0034] S04, calculate the running time T based on the time model sd , including the following sub-steps S04-1 to S04-3:

[0035] S04-1, Calculate the draft model M d The execution time T d :

[0036]

[0037] In the formula, α d ,γ d ,δ d For the draft model M d The performance coefficient on a given computing platform, N dc (i) and N db (i) are the draft models M d The number of context tokens and batch tokens in the i-th forward pass.

[0038] S04-2, calculate the target model M t The execution time T v :

[0039]

[0040] In the formula, p i is the length of the i-th draft sequence, α v ,γ v ,δ v The target model M t Performance coefficient on a given computing platform.

[0041] In the above steps S04-1 and S04-2, the draft model M d The execution time T d and the target model M t The execution time T v Involves multiple forward propagation calculations.

[0042] S04-3, calculate the running time T sd =T d +T v .

[0043] S05, using the service level target-aware efficiency estimator, calculate the throughput thpt under the current speculation length SL SL :

[0044] If T sd >tpot, then thpt SL =-1; if Tsd ≤tpot, then

[0045] Among them, T sd As obtained in step S04-3, tpot is the service level objective constraint, and NAT is obtained in step S03.

[0046] S06, determining the throughput thpt in step S04 SL The corresponding speculation length SL is:

[0047] (1) If SL=0, set SL=1 and return to step S03.

[0048] (2) If SL≠0, proceed to step S06.

[0049] S07, speculative decoding process:

[0050] (1) If SL≠1 or the speculative controller decides to perform speculative decoding, the process goes to step S08.

[0051] (2) If SL = 1 or the speculative controller decides not to perform speculative decoding, the input sequence set X is input into the target model M t After forward propagation calculation, the acceptance sequence set Z = {z1, ..., z BS} to update the input sequence set X and enter step S12, where z i =(z i,1 ) is the i-th accepted sequence.

[0052] Among them, the forward propagation calculation formula is as follows:

[0053]

[0054] Where M model represents the model that performs forward propagation, M model ∈{M d ,M t}, i = 1, ..., BS, p i is the length of the i-th draft sequence, the draft model M d and the target model M t have the same vocabulary size.

[0055] The steps of updating the input sequence set X are as follows S07-1 to S07-3:

[0056] S07-1, forward propagation is performed on each draft sequence to obtain the probability distribution P(z i,1 |x 1:n ).

[0057] S07-2, calculate the acceptance sequence set Z: zi =(z i,1 ),

[0058] S07-3, obtain the updated input sequence set X = {x1, ..., x BS}, x i =(x i ,z i ).

[0059] S08, input sequence set X = {x1,…,x BS} and the generated draft sequence set Y = {y1,…,y BS} The concatenated draft input sequence set {(x1,y1),…,(x BS ,y BS )} Input draft model M d After the forward propagation calculation, the draft sequence set Y is updated according to the output distribution and the greedy sampling standard, and its confidence is recorded and the historical average confidence of the word unit is calculated. Update, where y i =(y i,1 ,…,y i,SL-1 ) is the i-th draft sequence, i=1,…,BS, and specifically includes the following sub-steps S08-1 to S08-3:

[0060] S08-1, the draft sequence set Y is processed by the draft model M d The probability distribution P(y) calculated by forward propagation (the forward propagation calculation method is generally similar to that in step S07 and will not be repeated) 1,SL |x 1:n ,y 1:SL-1 ),…,P(y BS,SL |x 1:n ,y 1:SL-1 ), use the following calculation formula for greedy sampling:

[0061]

[0062] In the formula, y i,SL Indicates the draft model M d The prediction result of the SLth draft word in the i-th draft sequence, i=1,…,BS.

[0063] S08-2, y i,SL Splice to y i After that, y i =(y i,1 ,…,y i,SL-1 ,y i,SL ), complete the update of the draft sequence set Y.

[0064] S08-3, use the following calculation formula to update the historical average confidence

[0065]

[0066] c i,SL =max(P(y i,SL |x 1:n ,y 1:SL-1 ))

[0067] N=N+BS

[0068] Where N is the total number of historical draft tokens generated by speculative decoding.

[0069] S09, dynamically adjusting the speculation length SL, includes the following sub-steps S09-1 to S09-2:

[0070] S09-1, calculate the current predicted number of accepted words NAT and the throughput thpt under the current speculation length SL according to the method of step S03 and step S05. SL .

[0071] S09-2, the speculation controller uses the current speculation length to download thpt SL With thpt SL-1 Compare and determine whether to continue increasing the speculation length:

[0072] (1) If thpt SL >thpt SL-1 , it means that the increase of the current speculation length can improve the throughput, and the speculation controller will decide to continue to increase the speculation length, add 1 to the value of SL and return to step S03.

[0073] (2) If thpt SL ≤thpt SL-1 , it means that the increase of current speculation length cannot improve the throughput, and the speculation length has reached the optimal value. The optimal value SL * =SL-1.

[0074] S10, confidence priority verifier screening, including the following sub-steps S10-1 to S10-2:

[0075] S10-1, the confidence-prioritized verifier generates a draft sequence set Y = {y1, ..., y BS}, check the acceptance rate of each draft word one by one in the order of acceptance rate from low to high, and remove the word with the lowest current acceptance rate from the draft word set, where, is the i-th draft sequence.

[0076] S10-2, calculate the throughput thpt' after removing the word with the lowest current acceptance rate according to the method of steps S03 to S05 SL , if thpt' SL >thpt SL , then return to step S10-1, if not, then restore the last removed word unit and output the updated draft word unit set Y'={y'1,...,y' BS}, where y' i Represents the i-th draft sequence after elimination.

[0077] S11, update the input sequence set X:

[0078] The input sequence set X = {x1,…,x BS} and the draft sequence set Y'={y'1,…,y' BS}The concatenated efficient draft input sequence set {(x1,y'1),…,(x BS ,y' BS )} Input target model M t Perform forward propagation calculations, for each draft sequence Generate an acceptance sequence set Z = {z1,…,z BS} and update the input sequence set X.

[0079] Where i = 1,…,BS,p i is the length of the i-th draft sequence, q i is the accepted token length of the i-th draft sequence in Y'.

[0080] The steps for updating the input sequence set X are as follows S11-1 to S11-3:

[0081] S11-1, perform forward propagation calculation on each draft sequence (the forward propagation calculation method is generally similar to that in step S07 and will not be repeated here) to obtain a probability distribution

[0082] S11-2, calculate the acceptance sequence set Z:

[0083]

[0084] Where i = 1,…,BS, j = 1,…,p i .

[0085] S11-3, obtain the updated input sequence set X = {x1, ..., x BS}, x i =(x i ,zi ).

[0086] S12, remove the input sequence with the end mark of large model generation from the input sequence set X, add the newly received input sequence, if the input sequence set Z is not empty, enter step S02 to continue the speculative decoding process, if the input sequence set Z is empty, terminate the iteration.

[0087] This embodiment also provides a large model reasoning acceleration system based on adaptive speculative decoding, which uses a large model reasoning acceleration method based on adaptive speculative decoding in the embodiment, including an initialization module, a throughput evaluation module, a dynamic speculative controller, a confidence priority verifier, a draft model reasoning module and a target model reasoning module.

[0088] The initialization module is used to allow the user to set the initial historical average confidence of the draft word according to the method of steps S01 to S02. And the input sequence set X = {x1,…,x BS}Related parameters.

[0089] The throughput evaluation module is used to calculate the predicted number of accepted words NAT and the running time T according to the method of steps S03 to S05. sd And the throughput thpt SL .

[0090] The dynamic speculation controller is used to dynamically adjust the speculation length SL to the optimal value SL according to the method of steps S06, S07, and S09. * .

[0091] The confidence-prioritized verifier is used to screen draft word units with high acceptance rates according to the method of step S10.

[0092] The draft model reasoning module is controlled by the dynamic speculation controller to determine whether to update the draft sequence set Y according to the method of step S08.

[0093] The target model inference module is used to generate a verified output sequence and update the input data according to the method of steps S11 to S12.

[0094] <Test example>

[0095] This test example uses a large model reasoning acceleration system based on adaptive speculative decoding in the embodiment, and performs testing according to a large model reasoning acceleration method based on adaptive speculative decoding provided in the embodiment.

[0096] The test method is as follows: select Vicuna-160M+Vicuna-33b-v1.3 as the draft-target model combination as the application target of speculative decoding, select the Conversation production traces of three real production traces AzureLLMInferenceDataset2023 and 80 random data from each of the six data sets (MT-bench, WMT14 DE-EN, CNN / DailyMail, NaturalQuestions, GSM8K, DPR), and build a real large model inference scenario.

[0097] The large model inference acceleration method based on adaptive speculative decoding, autoregressive decoding ARD, speculative decoding SD1 with a preset speculative length of 1, speculative decoding SD3 with a preset speculative length of 3, speculative decoding SD5 with a preset speculative length of 5, threshold-based dynamic speculative decoding Threshold-based implemented by the transformer library, and the most advanced speculative decoding method SpecInfer in the embodiments are deployed on the same inference system vLLM, and the average large model inference delay of each system in the draft-target model combination and the real large model inference scenario is compared with the acceleration ratio of autoregressive decoding.

[0098] The average large model inference latency of each system compared to the speedup of autoregressive decoding is shown in Table 1 below:

[0099] Table 1 (Comparison of the speedup ratio of the method / system in this test example with several existing technologies)

[0100] Method Name Speedup ARD 1.00× SD1 1.07× SD3 0.83× SD5 0.33× Threshold-based 0.09× SpecInfer 0.09× This test case 1.27×

[0101] Functions and Effects of the Embodiments

[0102] According to the large model reasoning acceleration method and system based on adaptive speculative decoding involved in this embodiment, because the throughput is estimated by predicting the number of accepted words and the running time required for the current iteration, and then the speculative controller decides whether to perform speculative decoding, and the draft sequence set is updated through the draft model and the updated throughput is estimated until the speculative length reaches the optimal value for obtaining the optimal throughput, and then the high acceptance rate words are screened out by the confidence priority verifier, and the output sequence set of speculative decoding is obtained after verification by the target model. Therefore, the large model reasoning acceleration method and system based on adaptive speculative decoding of this embodiment can adaptively reduce the reasoning delay in various real large model reasoning scenarios.

[0103] Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A large model reasoning acceleration method based on adaptive speculative decoding, characterized in that: Efficient inference service for service level target awareness, low end-to-end latency, and high resource utilization includes the following steps: S01, set the historical average confidence of the draft word S02, according to the input sequence set X = {x1, ..., x BS }, set the batch size to BS and the speculation length SL = 0, where x represents the input sequence; S03, calculate the number of predicted accepted words NAT: In the formula, p i is the length of the i-th draft sequence, AR i,j represents the probability that the jth draft word in the i-th draft sequence is accepted, c i,k represents the confidence score of the kth draft word in the i-th draft sequence, c i,0 =1, S04, calculate the running time T based on the time model sd , where the running time T sd Including draft model M d The execution time T d and the target model M t The execution time T v ; S05, according to the running time T sd , service level objective constraint tpot and predicted number of accepted tokens NAT, use the service level objective-aware efficiency estimator to calculate the throughput thpt under the current speculation length SL SL ; S06, determine the throughput thpt SL The corresponding speculation length SL, if SL=0, set SL=1 and repeat steps S03 to S05 until SL≠0; S07, if SL≠1 or the speculative controller decides to perform speculative decoding, then enter step S08, if SL=1 or the speculative controller decides not to perform speculative decoding, then input the input sequence set X into the target model M t After forward propagation calculation, the acceptance sequence set Z = {z1, ..., z BS } to update the input sequence set X and enter step S12, Among them, z i =(z i,1 ) is the i-th accepted sequence; S08, input sequence set X = {x1,…,x BS } and the generated draft sequence set Y = {y1,…,y BS } The concatenated draft input sequence set {(x1,y1),…,(x BS ,y BS )} Input the draft model M d After the forward propagation calculation, the draft sequence set Y is updated according to the output distribution and the greedy sampling standard, and its confidence is recorded and the historical average confidence of the word unit is calculated. To update, where y i =(y i,1 ,…,y i,SL-1 ) is the i-th draft sequence, i=1,…,BS; S09, calculate the current predicted number of accepted words NAT and the throughput thpt under the current speculation length SL SL After that, according to the throughput thpt SL The speculation controller is used to adjust the speculation length SL and return to step S03 to iterate until the speculation length SL reaches the optimal value SL * ; S10, using the confidence priority verifier to filter out the currently generated draft sequence set Y = {y1, ..., y BS }, in, is the i-th draft sequence; S11, input sequence set X = {x1,…,x BS } and the draft sequence set Y'={y'1,...,y' BS }The concatenated efficient draft input sequence set {(x1,y'1),…,(x BS ,y' BS )} Input target model M t Perform forward propagation calculations, for each draft sequence Generate an acceptance sequence set z={z1,…,z BS } and update the input sequence set X, where i=1,…,BS,p i is the length of the i-th draft sequence, q i is the accepted word length of the i-th draft sequence in Y'; S12, remove the input sequence with the end mark of large model generation from the input sequence set X, add the newly received input sequence, if the input sequence set X is not empty, enter step S02 to continue the speculative decoding process, if the input sequence set X is empty, terminate the iteration.

2. The large model reasoning acceleration method based on adaptive speculative decoding according to claim 1 is characterized in that: in, In step S04: In the formula, α d ,γ d ,δ d For the draft model M d The performance coefficient on a given computing platform, N dc (i) and N db (i) are the draft models M d The number of context tokens and batch tokens during the i-th forward propagation, In the formula, p i is the length of the i-th draft sequence, α v ,γ v ,δ v The target model M t Performance coefficient on a given computing platform.

3. The large model reasoning acceleration method based on adaptive speculative decoding according to claim 1 is characterized in that: in, In step S05, if T sd >tpot, then thpt SL = -1; If T sd ≤tpot, then 4. The large model reasoning acceleration method based on adaptive speculative decoding according to claim 1, characterized in that: in, The forward propagation calculation formula is as follows: Where M model represents the model that performs forward propagation, M model ∈{M d ,M t }, i = 1, ..., BS, p i is the length of the i-th draft sequence, the draft model M d and the target model M t have the same vocabulary size.

5. The large model reasoning acceleration method based on adaptive speculative decoding according to claim 4 is characterized in that: in, In step S07, the steps of updating the input sequence set X are as follows: S07-1, forward propagation is performed on each draft sequence to obtain the probability distribution P(z i,1 |x 1:n ); S07-2, calculate the acceptance sequence set Z: z i =(z i,1 ), S07-3, obtain the updated input sequence set X = {x1, ..., x BS }, x i =(x i ,z i ).

6. The large model reasoning acceleration method based on adaptive speculative decoding according to claim 4 is characterized in that: in, Step S08 includes the following sub-steps: S08-1, the draft sequence set Y is processed by the draft model M d The probability distribution P(y 1,SL |x 1:n ,y 1:SL-1 ),…,P(y BS,SL |x 1:n ,y 1:SL-1 ), use the following calculation formula for greedy sampling: In the formula, y i,SL Indicates the draft model M d The prediction result of the SLth draft word in the i-th draft sequence, i=1,…,BS; S08-2, y i,SL Splice to y i After that, y i =(y i,1 ,…,y i,SL-1 ,y i,SL ), complete the update of the draft sequence set Y, S08-3, use the following calculation formula to update the historical average confidence c i,SL =max(P(y i,SL |x 1:n ,y 1:SL-1 )) N=N+BS Where N is the total number of historical draft tokens generated by speculative decoding.

7. The large model reasoning acceleration method based on adaptive speculative decoding according to claim 1 is characterized in that: in, In step S09, the speculation controller uses the current speculation length to set thpt SL With thpt SL-1 Compare and determine whether to continue increasing the speculation length: If thpt SL >thpt SL-1 , it means that the increase of the current speculation length can improve the throughput, and the speculation controller will decide to continue to increase the speculation length and increase the value of SL by 1. If thpt SL ≤thpt SL-1 , it means that the increase of current speculation length cannot improve the throughput, and the speculation length has reached the optimal value, the optimal value SL * =SL-1.

8. The large model reasoning acceleration method based on adaptive speculative decoding according to claim 1, Features: Wherein, step S10 includes the following sub-steps: S10-1, the confidence priority verifier generates a draft sequence set Y={y1,…,y BS }, check the acceptance rate of each draft word one by one in the order of acceptance rate from low to high, and remove the word with the lowest current acceptance rate from the draft word set; S10-2, calculate the throughput thpt' after removing the word with the lowest current acceptance rate SL , if thpt' SL >thpt SL , then return to step S10-1, if not, then restore the last removed word unit and output the updated draft word unit set Y'={y'1,...,y' BS }, where y' i Represents the i-th draft sequence after elimination.

9. The large model reasoning acceleration method based on adaptive speculative decoding according to claim 4 is characterized in that: in, In step S11, the steps of updating the input sequence set X are as follows: S11-1, forward propagation calculation is performed on each draft sequence to obtain the probability distribution S11-2, calculate the acceptance sequence set Z: Where i = 1,…,BS, j = 1,…,p i ; S11-3, obtain the updated input sequence set X = {x1, ..., x BS }, x i =(x i ,z i ).

10. A large model reasoning acceleration system based on adaptive speculative decoding, characterized in that: The large model reasoning acceleration method based on adaptive speculative decoding as described in any one of claims 1 to 9 is used, comprising: Initialization module, used for users to set the initial historical average confidence of draft word And the input sequence set X = {x1,…,x BS }Related parameters; Throughput evaluation module, used to calculate the predicted number of accepted tokens NAT and running time T sd And the throughput thpt SL ; Dynamic speculation controller, used to dynamically adjust the speculation length SL to the optimal value SL * ; Confidence-prioritized validator to filter high-acceptance draft tokens; A draft model reasoning module, which is controlled by the dynamic speculative controller to determine whether to update the draft sequence set Y; and The target model inference module is used to generate verified output sequences and update input data.

Images