Large language model reasoning optimization method based on cascade and speculative decoding strategy

Through the combination of cascading and inferred decoding strategies, small language models are called step by step in parallel to verify the output of large models, which solves the problems of high computing resources, slow inference speed and unstable generation quality of traditional large language models, and achieves efficient and stable inference The process reduces cost and time and improves the user experience.

CN119047579APending Publication Date: 2024-11-29PIO CLOUD COMPUTING (SHANGHAI) CO LTD

Patent Information

Application Number
CN202411116100.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In the inference process, traditional large language models have high demand for computing resources, slow inference speed, and unstable generation quality, resulting in high cost, low efficiency, poor user experience, and are susceptible to noise, which limits the popularization and application of the model.

Method used

Using cascading strategy and inferred decoding strategy, the draft text is generated through cascading strategy calling small language models of different parameter sizes step by step, and the inferred decoding strategy is used to verify and correct it in parallel on the big model, and combined with the loss function to optimize the model selection, reduce the calculation Cost and reasoning time to improve generation quality.

Benefits of technology

It significantly reduces computational cost and inference time, improves generation quality and inference speed, ensures flexible switching between different quality requirements and computing resources, and improves the efficiency and scalability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119047579A_ABST
    Figure CN119047579A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of language model reasoning, and discloses a large language model reasoning optimization method based on cascading and speculation decoding strategies, which comprises the following steps: S1, inputting cue words of a user to a small language model family, and quickly generating a plurality of draft text tokens by using a cascading strategy; s2, inputting a current text sequence into the large oracle model, verifying by using a speculative decoding strategy, cutting off a rejected text token, and generating a text output token of the large language model; s3, the step S1 and the step S2 are cycled until the large oracle model generates the result representing the end of the generation task; gt, endgt; and a label. According to the method, a cascading strategy and a speculation decoding strategy are introduced, the advantages of the two strategies are combined, and a new speculation cascading technology is designed, so that the reasoning speed and efficiency are improved. The cascading strategy is formed by combining a plurality of language models with different levels and different parameter sizes, and the speculative decoding strategy is used for reducing waiting time by deducing in advance and generating part of results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of language model reasoning, and in particular to a large language model reasoning optimization method based on cascade and inference decoding strategies. Background Art

[0002] In the process of large language model (LLM) reasoning, traditional methods have some significant problems, mainly focusing on computing resource requirements, reasoning speed and generation quality.

[0003] First, the demand for computing resources is high. Traditional large language models often require a lot of computing resources during the reasoning process, especially when the model scale is large, which will lead to the following problems: high cost, requiring expensive hardware such as high-performance GPU or TPU, and high operating cost; high energy consumption, a large amount of computing resource consumption leads to high energy consumption, which has a negative impact on the environment; strong device dependence, only high-end equipment can meet the reasoning requirements, which limits the popularity and application scenarios of the model. Secondly, large models often require a long reasoning time when processing complex tasks, which will lead to the following problems: high latency, long reasoning time, poor user experience, especially in real-time application scenarios; low efficiency, low reasoning efficiency, processing speed cannot meet high concurrency requirements, affecting the scalability of the application; untimely response, in scenarios that require quick decision-making, it is impossible to provide results in time, affecting the quality of decision-making. Finally, the generation quality of traditional large language models may be unstable during the reasoning process, which is mainly reflected in the following aspects: inconsistent generated content. Different inputs may produce outputs of inconsistent quality, affecting user trust; lack of precision. When generating long texts, the model may deviate from the topic, resulting in inaccurate or irrelevant content; susceptible to noise. The model is very sensitive to noise and errors in the input data and may generate low-quality results.

[0004] These problems limit the widespread deployment and use of large language models in practical applications, and new methods and strategies are urgently needed to improve the reasoning efficiency and generation quality of the model. The present invention combines the cascade strategy and the speculative decoding strategy to solve these problems. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings existing in the above-mentioned prior art, and to provide a large language model reasoning optimization method based on cascade and speculative decoding strategies. In the language model reasoning process, the cascade strategy and the speculative decoding strategy are used in combination to improve the reasoning speed and efficiency. The cascade strategy combines multiple language models of different levels and parameter sizes, and the speculative decoding strategy reduces the waiting time by inferring and generating partial results in advance.

[0006] A large language model reasoning optimization method based on cascade and speculative decoding strategy includes the following steps:

[0007] S1: Input the user's prompt words into the small language model family and use the cascade strategy to quickly generate several draft text tokens;

[0008] S2: Input the current text sequence into the large oracle model, use the speculative decoding strategy for verification, truncate the rejected text tokens and generate a text output token of the large language model itself;

[0009] S3: looping steps S1 and S2 until the large prediction model generates a signal indicating that the generation task is completed. <end>Label.

[0010] Wherein, in step S1, the cascade strategy specifically includes:

[0011] The small language model family composed of multiple small language models is used at the same time, denoted as M q ={m1,…,m n }, these models are sorted from small to large according to parameter size, and each of the models generates different types of text sequences of different qualities.

[0012] Further, selectively calling the small language model from small to large specifically includes:

[0013] First, the user's prompt word is input, and the smallest model m1 generates an output γ text tokens. The quality of the output text sequence of each small language model is evaluated based on the confidence threshold rule. If the confidence is lower than a certain threshold α, the larger model m2 is called to regenerate the text sequence, and so on.

[0014] Furthermore, the required balance between cost and quality is achieved by adjusting the threshold, specifically including:

[0015] S11: Set a threshold α∈[0,1];

[0016] S12: For a given prefix x1…x t , call the small model m1 to generate the next text token x t+1 ;

[0017] S13: Check whether the probability of the token generated by m1 is less than 1-α, that is, m1(x t+1 |x1…x t )<1-α,;

[0018] If yes, use the larger model m2 to generate a new text token; otherwise, keep the m1 text token;

[0019] S14: The selection of the model is similar, and the above steps are repeated to generate γ draft text tokens x t+1 …x t+γ .

[0020] Furthermore, in steps S12 to S14, the Chow's rule dynamic adjustment mechanism is used to determine whether it is necessary to transfer from the current small model to a larger model, specifically including:

[0021] For a given prefix x1…x t , first calculate the smaller model m i The distribution of m i (·|x1…x t ), then checks whether the probability of the output token sequence is below a pre-selected threshold, and if so, calls the larger model m i+1 To generate the next token x t+1 ; Otherwise continue to use the smaller model m i Generated token x t+1 .

[0022] The Chow's rule dynamic adjustment mechanism is as follows:

[0023]

[0024] Among them, α is a cost parameter. The higher the α value, the lower the frequency of transfer to the larger model p. Chow (x0…x t )=1 indicates inference x t+1 The task is transferred to the next larger model, w represents a word, Represents the set of all words, m i (w|x0…x t ) represents the model m i The probability of generating the word w is determined by selecting a threshold based on different confidence metrics.

[0025] Further, in step S2, the inferential decoding strategy is used in parallel in the large model M p Verify the draft text output on x t+1 …x t+γ , is the large model M p Rejected tokens are replaced, and tokens are resampled using the revised distribution.

[0026] Furthermore, the use of the speculative decoding strategy is performed in parallel on the large model M p Verify the text output on x t+1 …x t+γ The specific steps include:

[0027] S21: In the large model M p Run these γ text tokens x in parallel t+1 …x t+γ , when generating the next token, check all tokens in the sequence and calculate the probability of each token in the sequence to verify the sequence x generated by the small model t+1 …x t+γ .

[0028] S22: Verify small model family M q Is the generated token acceptable? If there is a token that is not accepted, roll back to the first rejected token t+τ and use the large model M p Generate a new token replacing x t+τ , and with x0…x t+τ As a new prefix starting from the cascade strategy using the small model family M q Generate new γ text outputs.

[0029] Further, the large model M p Rejected tokens are replaced and tokens are resampled using the revised distribution:

[0030] In the verification phase, the small model family M q Generated draft text token x t+τ Accept with the following probability:

[0031]

[0032] If rejected, they are replaced with new tokens sampled from the revised new distribution:

[0033] x t+τ ~norm(max{0,M p (x t+τ |x1…x t+τ-1 )-M q (x t+τ |x1…x t+τ-1 )}), where norm represents a regularization operation with a sum of 1. The sampling process is theoretically equivalent to sampling from the prefix x0…x t In the autoregressive way from the large model M p If all draft text tokens are accepted, then the large model M p Generate a new token x t+γ+1 , and then continue to generate text sequences in a loop starting from the cascade strategy.

[0034] Further, the task of transferring to the large model M is reduced. p The expected loss of generating tokens includes:

[0035] For a fixed prefix x1…x t , using a loss function of the following form:

[0036]

[0037] Among them, δ ≥ 0 is the cost penalty for switching to a larger model, loss is the loss function, and logarithmic loss loss is used. log (x t ,M q )=-log(M q (x1…x t ), the optimal cascade transition rule is expressed as:

[0038]

[0039] Compare the expected loss from the small model q with the expected cost of calling the large model p, and decide to postpone if the latter is smaller. In order to approximate the optimal cascading rule during inference, use the model's confidence estimate instead of the expected loss.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The present invention adopts a cascade strategy of multi-level model calling and dynamic selection. This multi-level model calling strategy allows flexible switching between different quality requirements and computing resources. By gradually increasing the model size, it ensures that the task is completed without calling a large model, thereby significantly reducing the computing cost and inference time.

[0042] (2) The present invention can greatly speed up the inference speed by using multiple small model families to process the initially generated text sequence in parallel, while speculating the decoding strategy to verify the draft text output in parallel on the large model, further reducing the inference time;

[0043] (3) The present invention optimizes by using a loss function and combines the Chow's rule dynamic adjustment mechanism to select a suitable model to ensure that the expected loss is minimized during the generation process, thereby improving the overall quality of the generated text. At the same time, the inferred decoding strategy allows error compensation for the draft text in the verification phase, and corrects the generation errors by resampling tokens from the corrected distribution, further improving the generation quality;

[0044] (4) The present invention adopts parallel verification and sampling optimization of the inference decoding strategy, and greatly reduces the computational overhead of each step in the reasoning process through parallel processing and residual sampling optimization. Even if the large model verifies and corrects the draft text, the overall computational complexity is still greatly reduced, which significantly improves the reasoning speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0046] Figure 1 A flow chart of a large language model reasoning optimization method based on cascade and speculative decoding strategies of the present invention;

[0047] Figure 2 This is a reasoning process framework diagram of a large language model reasoning optimization method based on cascade and speculative decoding strategies of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention means the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0050] The present invention combines the advantages of these two strategies by designing a new speculative cascade technology. We conducted benchmark language task experiments on the T5 model. The experimental results show that the method of the present invention is significantly better than the traditional cascade and speculative decoding methods in terms of cost-quality trade-off. Specifically, the present invention not only significantly improves the inference speed, but also greatly reduces the computational cost while maintaining the output quality.

[0051] The specific implementation of the present invention is described below with reference to the accompanying drawings and embodiments.

[0052] Example 1

[0053] like Figure 1 As shown, the technical solution of a large language model reasoning optimization method based on cascade and inference decoding strategy provided by this embodiment includes the following steps:

[0054] S1: Input the user's prompt words into the small language model family and use the cascade strategy to quickly generate several draft text tokens;

[0055] S2: Input the current text sequence into the large oracle model, use the speculative decoding strategy for verification, truncate the rejected text tokens and generate a text output token of the large language model itself;

[0056] S3: looping steps S1 and S2 until the large prediction model generates a signal indicating that the generation task is completed. <end>Label.

[0057] The specific reasoning framework is as follows Figure 2 As shown, first, in step S1, we use a cascade strategy to balance the quality and latency of the large language model. Specifically, we use a small model family M consisting of multiple small models at the same time. q ={m1,…,m n }, these models are sorted from small to large according to parameter size, and each model can generate different types of text sequences with different qualities. We find a balance between output text quality and latency by selectively calling these models from small to large.

[0058] In this embodiment, for the user's prompt word, the smallest model m1 first generates γ output text tokens, and the quality of the output text sequence of each small language model is evaluated based on the confidence threshold rule. If the confidence is lower than a certain threshold α, the larger model m2 is called to regenerate the text sequence, and so on. By adjusting this threshold, a balance between the required cost and quality can be achieved.

[0059] Specifically, achieving a desired balance between cost and quality by adjusting the threshold includes:

[0060] S11: Set a threshold α∈[0,1]. The higher the value of α, the lower the frequency of transfer to the larger model p.

[0061] S12: For a given prefix x1…x t , call the small model m1 to generate the next text token x t+1 ;

[0062] S13: Check whether the probability of the token generated by m1 is less than 1-α, that is, m1(x t+1 |x1…x t )<1-α,;

[0063] If yes, use the larger model m2 to generate a new text token; otherwise, keep the m1 text token;

[0064] S14: The selection of the model is similar, and the above steps are repeated to generate γ draft text tokens x t+1 …x t+γ .

[0065] In steps S12 to S14, the Chow's rule dynamic adjustment mechanism is used to determine whether it is necessary to transfer from the current small model to a larger model, specifically including:

[0066] For a given prefix x1…x t , first calculate the smaller model m i The distribution of m i (·|x1…x t ), then checks whether the probability of the output token sequence is below a pre-selected threshold, and if so, calls the larger model m i+1 To generate the next token x t+1 ; Otherwise continue to use the smaller model m i Generated token x t+1 .

[0067] The Chow's rule dynamic adjustment mechanism is as follows:

[0068]

[0069] Among them, α is a cost parameter. The higher the α value, the lower the frequency of transfer to the larger model p. Chow (x0…x t )=1 indicates inference x t+1 The task is transferred to the next larger model, w represents a word, w represents the set of all words, m i (w|x0…x t ) represents the model m i The probability of generating the word w is determined by selecting a threshold based on different confidence measures, such as the entropy of the model probability distribution.

[0070] Subsequently, we use the speculative decoding strategy to perform parallel decoding on the large model M p Verify the draft text output on x t+1 …x t+γ , is the large model M p Rejected tokens are replaced and tokens are resampled using the revised distribution. This approach is theoretically equivalent to directly sampling from the larger model M p The proposed method samples tokens in , but has lower computational cost.

[0071] In this embodiment, the speculative decoding strategy is used in parallel in the large model M p Verify the text output on x t+1 …x t+γ The specific steps include:

[0072] S21: In the large model M p Run these γ text tokens x in parallel t+1 …x t+γ , although these LLM models usually generate one word at a time, they can process multiple tokens at a time. When generating the next token, all tokens in the sequence are checked and the probability of each token in the sequence is calculated to verify the sequence x generated by the small model. t+1 …x t+γ .

[0073] S22: Verify small model family M q Is the generated token acceptable? If there is a token that is not accepted, roll back to the first rejected token t+τ and use the large model M p Generate a new token replacing x t+τ , and with x0…x t+τ As a new prefix starting from the cascade strategy using the small model family M q Generate new γ text outputs.

[0074] Among them, in the verification stage, the small model family M q Generated draft text token x t+τ Accept with the following probability:

[0075]

[0076] If rejected, they are replaced with new tokens sampled from the revised new distribution:

[0077] x t+τ ~norm(max{0,M p (x t+τ |x1…x t+τ-1 )-M q (x t+τ |x1…x t+τ-1 )}),

[0078] Where norm represents a regularization operation with a sum of 1. The sampling process is theoretically equivalent to sampling from the prefix x0…x t In the autoregressive way from the large model M p If all draft text tokens are accepted, then the large model M p Generate a new token x t+γ+1 , and then continue to generate text sequences in a loop starting from the cascade strategy.

[0079] Specifically, in this embodiment, we use a lossy variant that allows for some deviation from the validator distribution p. In this case, the draft token x t+τ Accept with the following probability:

[0080]

[0081] Here, α∈[0,1) is a strictness parameter, and higher values ​​indicate greater deviation from the output of the large model p. Rejected tokens can be replaced with new tokens sampled from the residual distribution:

[0082]

[0083] Ultimately, our goal is to reduce the amount of time it takes to push the task to the large model M p The expected loss caused by generating tokens. Specifically including:

[0084] For a fixed prefix x1…x t , using a loss function of the following form:

[0085]

[0086] Among them, δ ≥ 0 is the cost penalty for switching to a larger model, loss is the loss function, and logarithmic loss loss is used. log (x t ,M q )=-log(M q (x1…x t ), the optimal cascade transition rule is expressed as:

[0087]

[0088] Compare the expected loss from the small model q with the expected cost of calling the large model p, and decide to postpone if the latter is smaller. In order to approximate the optimal cascading rule during inference, use the model's confidence estimate instead of the expected loss.

[0089] In this embodiment, the cascade strategy used is a technology widely used in various fields (such as machine learning, computer vision and signal processing), which aims to process tasks step by step in stages to improve efficiency and performance. By breaking down complex tasks into multiple simple and sequentially executed subtasks, the cascade strategy can effectively reduce the use of computing resources while improving processing speed and accuracy. The key features and steps of the cascade strategy are staged processing: complex tasks are broken down into a series of simpler subtasks. Each subtask is processed in an independent stage, usually only a part of the overall data needs to be processed.

[0090] Sequential execution: Subtasks are executed in order, with the output of the previous stage serving as the input of the next stage. This sequential processing ensures that the problem size is gradually reduced at each stage, gradually approaching the final goal. Screening and filtering: At each stage, the complexity of the data set or task is reduced, usually by screening and filtering irrelevant or unimportant information. This reduces the computational burden of subsequent stages and improves overall efficiency.

[0091] The speculative decoding strategy is a decoding technique used in natural language processing models (such as language models or translation models) to speed up the process of generating sequences (such as sentences or paragraphs). It generates multiple candidate sequences and predicts their likelihood, thereby deciding the most likely result in advance to improve generation speed and efficiency. Key concepts and working principles Candidate sequence generation: Speculative decoding first generates multiple possible candidate sequences. These candidate sequences can be generated through multiple forward passes of the model, representing multiple outputs that the model may generate in different situations. Parallel prediction: These candidate sequences are usually generated in parallel to make full use of computing resources. With the help of parallel computing, multiple candidate sequences can be generated in a shorter time. Advance decision: After generating candidate sequences, the model evaluates and ranks these sequences and selects the most likely sequence. By deciding the most likely sequence in advance, the model can avoid unnecessary calculations, thereby speeding up the entire decoding process.

[0092] Finally, it should be noted that the above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

[0093] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.< / end> < / end>

Claims

1. A large language model reasoning optimization method based on cascade and speculative decoding strategy, characterized in that: The steps include: S1: Input the user's prompt words into the small language model family and use the cascade strategy to quickly generate several draft text tokens; S2: Input the current text sequence into the large oracle model, use the speculative decoding strategy for verification, truncate the rejected text tokens and generate a text output token of the large language model itself; S3: looping steps S1 and S2 until the large prediction model generates a signal indicating that the generation task is completed. <end> Label.< / end> 2. The large language model reasoning optimization method based on cascade and speculative decoding strategy according to claim 1 is characterized in that: In step S1, the cascade strategy specifically includes: The small language model family composed of multiple small language models is used at the same time, denoted as M q ={m1,…,m n }, these models are sorted from small to large according to parameter size, and each of the models generates different types of text sequences of different qualities.

3. The large language model reasoning optimization method based on cascade and speculative decoding strategy according to claim 2 is characterized in that: By selectively calling the small language model from small to large, specifically including: First, the user's prompt word is input, and the smallest model m1 generates an output γ text tokens. The quality of the output text sequence of each small language model is evaluated based on the confidence threshold rule. If the confidence is lower than a certain threshold α, the larger model m2 is called to regenerate the text sequence, and so on.

4. The large language model reasoning optimization method based on cascade and speculative decoding strategy according to claim 3 is characterized in that: The thresholds are adjusted to achieve the desired balance between cost and quality, including: S11: Set a threshold α∈[0,1]; S12: For a given prefix x1…x t , call the small model m1 to generate the next text token x t+1 ; S13: Check whether the probability of the token generated by m1 is less than 1-α, that is, m1(x t+1 |x1…x t )<1-α,; If yes, use the larger model m2 to generate a new text token; otherwise, keep the m1 text token; S14: The selection of the model is similar, and the above steps are repeated to generate γ draft text tokens x t+1 …x t+γ .

5. The large language model reasoning optimization method based on cascade and speculative decoding strategy according to claim 4 is characterized in that: In steps S12 to S14, the Chow's rule dynamic adjustment mechanism is used to determine whether it is necessary to transfer from the current small model to a larger model, specifically including: For a given prefix x1…x t , first calculate the smaller model m i The distribution of m i (·|x1…x t ), then checks whether the probability of the output token sequence is below a pre-selected threshold, and if so, calls the larger model m i+1 To generate the next token x t+1 ; Otherwise continue to use the smaller model m i Generated token x t+1 .

6. According to the large language model reasoning optimization method based on cascade and speculative decoding strategy of claim 5, the Chow's rule dynamic adjustment mechanism is expressed as follows: in, α is a cost parameter. The higher the value of α, the lower the frequency of transfer to the larger model p. Chow (x0…x t )=1 indicates inference x t+1 The task is transferred to the next larger model, w represents a word, Represents the set of all words, m i (w|x0…x t ) represents the model m i The probability of generating the word w is determined by selecting a threshold based on different confidence metrics.

7. The large language model reasoning optimization method based on cascade and speculative decoding strategy according to claim 4 is characterized in that: In step S2, the speculative decoding strategy is used in parallel in the large model M p Verify the draft text output on x t+1 …x t+γ , is the large model M p Rejected tokens are replaced, and tokens are resampled using the revised distribution.

8. The large language model reasoning optimization method based on cascade and speculative decoding strategy according to claim 7 is characterized in that: The speculative decoding strategy is used in parallel on the large model M p Verify the text output on x t+1 …x t+γ The specific steps include: S21: In the large model M p Run these γ text tokens x in parallel t+1 …x t+γ , when generating the next token, check all tokens in the sequence and calculate the probability of each token in the sequence to verify the sequence x generated by the small model t+1 …x t+γ . S22: Verify small model family M q Is the generated token acceptable? If there is a token that is not accepted, roll back to the first rejected token t+τ and use the large model M p Generate a new token replacing x t+τ , and with x0…x t+τ As a new prefix starting from the cascade strategy using the small model family M q Generate new γ text outputs.

9. The large language model reasoning optimization method based on cascade and speculative decoding strategy according to claim 8, characterized in that: The large model M p Rejected tokens are replaced and tokens are resampled using the revised distribution: In the verification phase, the small model family M q Generated draft text token x t+τ Accept with the following probability: If rejected, they are replaced with new tokens sampled from the revised new distribution: x t+τ ~norm(max{0,M p (x t+τ |x1…x t+τ-1 )-M q (x t+τ |x1…x t+τ-1 )}), Where norm represents a regularization operation with a sum of 1. The sampling process is theoretically equivalent to sampling from the prefix x0…x t In the autoregressive way from the large model M p If all draft text tokens are accepted, then the large model M p Generate a new token x t+γ+1 , and then continue to generate text sequences in a loop starting from the cascade strategy.

10. The large language model reasoning optimization method based on cascade and speculative decoding strategy according to claim 9, characterized in that: Reduce the task of shifting it to the large model M p The expected loss of generating tokens includes: For a fixed prefix x1…x t , using a loss function of the following form: Among them, δ ≥ 0 is the cost penalty for switching to a larger model, loss is the loss function, and logarithmic loss loss is used. log (x t ,M q )=-log(M q (x1…x t ), the optimal cascade transition rule is expressed as: Compare the expected loss from the small model q with the expected cost of calling the large model p, and decide to postpone if the latter is smaller. In order to approximate the optimal cascading rule during inference, use the model's confidence estimate instead of the expected loss.

Citation Information

Patent Citations

  • Large language model reasoning acceleration method and related device

    CN118333172A

  • Big language model reasoning method and device, equipment and storage medium

    CN118364918A

Cited By

  • Prospective sampling method and device based on draft model token screening

    CN119940548A

  • Information processing method and device, equipment and storage medium

    CN120234458A

  • Information processing method and device, equipment and storage medium

    CN120234458B

  • Method, device and equipment for detecting output error of large language model and medium

    CN120338103A

  • Response text generation method and device, electronic equipment and storage medium

    CN120354951A