Method and system for accelerating large model reasoning by using dynamic tree structure speculative decoding

Abstract

The invention relates to a method and a system for accelerating large model reasoning by using dynamic tree structure speculative decoding. The method comprises the following steps: processing input characters by adopting a small model to generate prediction characters of a tree structure; inputting the input characters and the prediction characters generated by the small model into the large model, obtaining an output result of the large model through a large model reasoning process, and judging whether the prediction characters generated by the small model are received or not, so as to obtain a finally generated text. According to the tree reasoning structure, the branch nodes with higher expected receiving probability can be utilized, and more tokens are expected to be generated while the speculation length is increased, so that the purpose of accelerating large model reasoning is achieved; the design of the dynamic determination ending condition can be ended more quickly when the method is poor in performance, and the method has better comprehensive performance compared with similar methods.

Description

Technical Field

[0001] The present invention belongs to the fields of information technology and artificial intelligence technology, and specifically relates to a method and system for accelerating large model reasoning by using dynamic tree structure speculative decoding. Background Art

[0002] Speculative decoding is a technology that accelerates the reasoning of large models and improves the performance of language model generation. When generating text, it predicts multiple possible subsequent words and dynamically selects the most appropriate output. This can improve the fluency and relevance of the generated text. A large model refers to a language model with a large training scale that can learn rich language knowledge. Such a large model can be applied to a variety of natural language processing tasks and has good performance in a variety of natural language processing tasks. However, due to the autoregressive process in the reasoning process, the large model cannot fully utilize the parallel acceleration capabilities of the existing general-purpose graphics processing unit (GPGPU) during the normal decoding process. Speculative decoding technology uses a relatively small-scale model with relatively poor performance to quickly generate a series of subsequent results, and then the large model verifies this series of predicted results in parallel to decide whether to accept these tokens. In this way, each time the large model processes a batch of results, it can fully utilize the computing power of the general-purpose graphics processing unit. At the same time, if the capabilities of the small model are close enough to those of the large model, the loss of prediction failure caused by this technology will be reduced to a level that can be compensated by the full utilization of computing power, achieving an end-to-end acceleration effect.

[0003] Existing methods represented by Sequoia proposed static tree-structured speculative reasoning methods based on the initial chain speculative decoding. These methods first predict multiple candidate results for the current result at one time. Each point predicts multiple candidate results to form a tree-structured inference result. Figure 1 As shown, Figure (a) is the original chain speculative reasoning method, Figure (b) is the k-chain speculative reasoning method based on this, and Figure (c) is a more general tree-structured speculative reasoning method. A series of methods such as Red Shirt assume that for each node, the final reception of its next prediction is independent and identically distributed. Further, they proposed the concept of reception rate vector, that is, the statistical situation of the reception rate of the large model generation results of the target task and the generation results of the small model selected by speculative decoding under this set of methods, and then estimated the reception situation of several subsequent forks of the same node in an average sense. Then, based on this prediction, the expected estimate of the final number of generated tokens for each node is estimated. Finally, the optimal prediction tree structure is calculated based on the aforementioned estimation.

[0004] The problem with existing chained speculative decoding technology is that, according to the principle of speculative decoding technology, the probability of receiving each element in the speculated token sequence is independent, and the prerequisite for the next token to be received is that the previous token is also received. Therefore, as the length of a single speculation increases, the expected return of the chained method decreases exponentially, resulting in a short effective speculation length in practice. However, the current mainstream large model technology is deployed on general-purpose graphics processors (GPGPUs) with high multi-core throughput and relatively lower single-core performance. The shorter speculation length makes it impossible to fully utilize the parallel processing capabilities of the general-purpose graphics processor in the subsequent processing, resulting in performance loss.

[0005] In addition, there are also some speculative decoding techniques that use tree structures. However, the problem with these techniques is that they need to obtain the probability distribution of the generation results of the large model in advance, and only when the probability distribution is assumed to be the same can the corresponding tree structure inference results be generated. In addition, the inference results of these methods remain unchanged after being obtained in advance. These methods assume that the situation faced by the large model when generating text at any time is the same, and will not make corresponding adjustments based on the different situations encountered in the actual generation. This assumption is seriously inconsistent with the actual situation, so it can only perform well in specific scenarios and is not applicable to a wider range of situations. Summary of the invention

[0006] In view of the above problems, the present invention provides a method and system for accelerating large model reasoning by using dynamic tree structure speculative decoding.

[0007] The technical solution adopted by the present invention is as follows:

[0008] A method for accelerating large model reasoning using dynamic tree structure speculative decoding, comprising the following steps:

[0009] Use a small model to process the input text and generate predicted characters in a tree structure;

[0010] The input text and the predicted characters generated by the small model are input into the large model together, the output result of the large model is obtained through the large model reasoning process, and it is determined whether to accept the predicted characters generated by the small model, so as to obtain the final generated text.

[0011] Furthermore, before reaching the prediction end condition, the small model continuously predicts new characters to be added to the tree structure; the small model processes each expandable node of the currently generated tree structure, and obtains the probability distribution of each expandable node generating the next character through the standard small model reasoning process; the probability of each character output by the small model is directly used as an estimate of the probability that the character will eventually be received by the large model, that is, the acceptance rate.

[0012] Furthermore, when the small model reaches the exit condition of the prediction tree structure, the generation of the current prediction tree structure is terminated; the exit condition is set as the size of the tree structure reaching a threshold, or is set as a threshold of the estimated reception rate of the predicted characters in the tree structure.

[0013] Furthermore, the optimal tree structure obtained by the small model is the tree structure with the maximum sum of the reception rates of all nodes under a given number of nodes in the tree; the predicted tree structure is efficiently obtained using the priority queue algorithm, and the priority queue algorithm includes: adding all candidate nodes to the same priority queue to ensure that the node popped up each time is the node with the highest reception rate in the current queue; after each node is popped up, a new candidate node is updated and added to the priority queue; the new candidate node can be an extended node, including a child node and a sibling node, and the child node is received only if the current node is received, and the sibling node is received only if the current node is not received.

[0014] Furthermore, the large model determines whether to accept the predicted characters generated by the small model, including: comparing the probability distribution of subsequent nodes obtained by the large model with the probability distribution of subsequent nodes predicted by the small model, determining whether the results generated by the small model match the results generated by the large model, and then deciding whether to accept the results of the small model.

[0015] Furthermore, the large model determines whether to receive the predicted character generated by the small model, including:

[0016] Map each random event corresponding to the probability distribution of the small model and each random event corresponding to the probability distribution obtained by the large model in a fixed rule;

[0017] After completing the mapping of random events, each random event of the small model will be in two states: mapped to the same event in the large model / mapped to different events in the large model; if the character sampled by the small model from the probability distribution belongs to the "mapped to the same event in the large model" state, it means that the large model has verified the character and the character will be accepted, otherwise the character will not be accepted.

[0018] A system for accelerating large model reasoning using dynamic tree structure speculative decoding, comprising:

[0019] A small model prediction module is used to process the input text using a small model to generate predicted characters in a tree structure;

[0020] The large model inference module is used to input the input text and the predicted characters generated by the small model into the large model, obtain the output result of the large model through the large model inference process, and determine whether to accept the predicted characters generated by the small model, so as to obtain the final generated text.

[0021] The beneficial effects of the present invention are as follows:

[0022] The tree-shaped reasoning structure of the present invention can utilize some forked nodes with higher expected reception probabilities, and expect to generate more tokens (characters) while increasing the inference length, thereby achieving the purpose of accelerating the reasoning of large models.

[0023] The method proposed in the present invention will dynamically determine the prediction tree structure according to each step of the output of the small model, rather than just determining what characters each node in the fixed tree structure specifically represents. Therefore, it is aware of the current scene and does not require prior knowledge of the approximate distribution of the content to be generated. It is more operational and more in line with the actual situation. In addition, the present invention will adjust the output probability distribution of the small model according to the previous prediction generation results (the ratio of the number of received characters to the predicted maximum depth) during the reasoning process (corresponding to the empirical value temperature in the specific implementation method, a large ratio indicates that the two models are closer, and the temperature should be lowered to make the output of the small model more concentrated, and vice versa, the output of the small model should be more divergent to make it easier to hit the results of the large model).

[0024] The design of dynamically determining the end condition of the present invention can end the process more quickly when the method performs poorly, and has better overall performance than similar methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of existing speculative reasoning methods.

[0026] Figure 2 It is a flow chart of the steps of the method of the present invention. DETAILED DESCRIPTION

[0027] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below through specific embodiments and drawings.

[0028] The present invention uses an appropriate small model to accelerate the reasoning speed of the large model by a speculative decoding method that dynamically generates a prediction tree structure. For the input instruction text, the large model is usually directly allowed to process the prefix and then output the subsequent text one by one. The autoregressive decoding of the large model results in the need to reprocess all previous text before each new character is generated, which greatly reduces the generation efficiency of the large model. The present invention uses a faster small model to first process the input text, generate possible subsequent characters in a tree structure, and then input them together to the large model so that the large model can determine which characters to receive in the end. In this way, the large model can receive more new characters while processing the same scale of text. When the running cost of the small model is negligible relative to the large model, the present invention achieves the acceleration of the reasoning of the large model by allowing the large model to generate more characters at a time.

[0029] In the present invention, small model and large model are relative concepts. The large model is a model with good performance that is actually used in the original application scenario. As long as the reasoning time of a model is significantly faster than the large model, it can be used as the small model in the present invention. For example, if llama2-70b is selected as the target model, in a scenario where the video memory is limited and the model cannot be completely stored, the model can only be stored in the memory, and the used part is read into the video memory each time. At this time, the overhead caused by moving data back and forth becomes the bottleneck of model reasoning. At this time, llama2-7b, which is closer to the target model but has a smaller video memory requirement and a significantly faster reasoning speed, can be used as a small model to accelerate the reasoning speed of the target 70b model.

[0030] The technical innovation of the present invention is mainly reflected in the method of generating a tree structure with a small model. The specific steps are:

[0031] 1. The small model maintains a tree structure of predicted characters that has been generated, and continuously predicts new characters and adds them to the tree structure before reaching the prediction end condition. The characters are nodes in the tree structure.

[0032] 2. Use the small model to process each expandable node of the currently generated tree structure, and obtain the probability distribution of the next character generated by each expandable node through the standard small model reasoning process. In chain-type speculative reasoning, if you want to increase the number of characters obtained each time by predicting more characters, you can only generate from the end of the current prediction node chain. In the speculative reasoning of the tree structure, you can continue to generate downward along any path starting from the root in the tree structure (expanding leaf nodes), and you can also expand a forked path at a certain position on this path (expanding sibling nodes). These two types of nodes are collectively referred to as "expandable nodes."

[0033] 3. Through experiments, it is found that in the speculative reasoning scenario, the probability distribution of the next character obtained by the small model is positively correlated with the probability of the character being eventually accepted by the target model. In practice, for convenience, the probability of each character output by the small model is directly used as an estimate of the probability that the character will eventually be accepted by the large model (i.e., the acceptance rate). The acceptance rate here refers to the probability that the character will eventually be generated by the large model. This probability comes from the sampling introduced by the mainstream large model technology in the decoding stage. The probability distribution obtained by the small model is generally obtained by calculating the output of the small model (logit), which is usually a probability distribution. Where temperature is a non-negative number, generally an empirical value of 0 to 2. This value can also be adjusted dynamically. The present invention finds that when the temperature is high, the predicted probability distribution tends to be flat, and is more inclined to obtain a wider prediction tree structure; when the temperature is low, the predicted probability distribution tends to be steep, and is more inclined to obtain a narrower and deeper prediction tree structure. Therefore, some result feedback in the actual execution process can be used to adjust the setting of the temperature value, such as the ratio of the actual number of received tokens in the previous execution results to the depth of the prediction tree. When this ratio is very small, it means that the deep prediction nodes are rarely accepted. At this time, the temperature value can be reduced to obtain a wider and shorter prediction tree to reduce invalid calculations. On the contrary, it means that the prediction tree is very accurate. The temperature can be increased to predict longer tokens to increase the number of tokens generated at a single time. Logit is the output of the small model, which is an n-dimensional floating point vector, where n is the size of the vocabulary, i represents the i-th token, and prob i is the obtained probability distribution, expressed as prob i The denominator in the above formula is a normalization to ensure that all prob i The sum of is 1, j is the subscript of the sum. According to this estimate prob i The node with the largest reception rate among all candidate nodes (i.e., expandable nodes) is selected and added to the predicted tree structure.

[0034] 4. The exit condition can be set when the size of the tree structure reaches a threshold (e.g. Figure 1 The N value in ( ) can also be set as a threshold value of the estimated acceptance rate of the predicted characters in the tree structure. When the exit condition of the prediction tree structure is reached, the generation of the current prediction tree structure is terminated.

[0035] According to the implementation method proposed in the previous text, for each node, its receiving rate is obtained, so the optimal tree structure is the tree structure with the maximum sum of the receiving rates of all nodes under the given number of nodes in the tree. Since the receiving rate of each node is a real number between 0 and 1, it can be guaranteed that the receiving rate of each node must be greater than the receiving rate of the expandable node brought by this node. In order to obtain the predicted tree structure more efficiently, the present invention uses the priority queue method to add all candidate nodes to the same priority queue. The priority queue algorithm will ensure that the node that pops up each time is the node with the largest receiving rate in the current queue. After each node pops up, a new candidate node is updated and added to the priority queue. The new candidate node is the expandable node (child node and brother node) mentioned in the previous text. The premise for the child node to be received is that the current node is received, so the (cumulative) receiving rate of this type of candidate node is the (cumulative) receiving rate of the parent node multiplied by the (independent) receiving rate of the current node. The premise for the brother node to be received is that the current node is not received. Therefore, the (cumulative) reception rate of this type of candidate nodes is the (cumulative) reception rate of the ancestor node multiplied by [1-the (independent) reception rate of the brother node] multiplied by the (independent) reception rate of the current node. The independent reception rate here is the reception rate when only this node is considered, and the cumulative reception rate is the probability of being received from the root node to the current node. In this way, it can be ensured that the optimal candidate node is obtained each time. Since the newly generated candidate node must be expanded from the currently selected node, and its expected reception rate will be compounded with a probability value not exceeding 1 when expanding, the expected reception rate of the candidate node newly added to the priority queue must be monotonically decreasing, so the method of the present invention obtains the optimal tree structure.

[0036] After obtaining the tree structure predicted by the small model, the generated tree structure prediction results are converted into a format that can be processed by the large model, and the output results are obtained through the standard large model reasoning process, and then verified to decide which predicted characters to receive.

[0037] After processing the predicted results, the large model will obtain the probability distribution of each subsequent node, that is, after inputting the predicted results to the large model during the reasoning process, the large model will obtain the logit corresponding to each token in the input, and then calculate the probability distribution, and then sample the final token based on the probability distribution. The present invention compares the probability distribution of the subsequent nodes obtained by the large model with the probability distribution of the subsequent nodes predicted by the small model to determine whether the results generated by the small model match the results generated by the large model, and then decides whether to receive the results of the small model. After determining which results generated by the small model are to be received by the large model, the final output result of the large model is obtained. And the previous judgment process also ensures that the results obtained by this method are the same distribution as the results directly inferred by the large model.

[0038] The specific process of the large model judging the prediction results of the small model is as follows:

[0039] 1) The prediction results of the small model are sampled from a probability distribution. After obtaining the probability distribution of the large model's prediction results, each random event corresponding to the probability distribution of the small model can be mapped to each random event corresponding to the probability distribution obtained by the large model using a fixed rule. The fixed rule is to avoid the impact of changes in the mapping rule on the acceptance rate, which would destroy the constraint that the output results are directly distributed with the large model.

[0040] 2) After completing the mapping of random events, each random event of the small model will be in two states: mapped to the same event in the large model / mapped to different events in the large model. If the character sampled by the small model from the probability distribution belongs to the "mapped to the same event in the large model" state, it means that the large model has verified the character, so the character will be accepted. Otherwise, it means that it will not be accepted. At this time, it is necessary to resample a new character from all events where the large and small models are mapped to different events as the output result of the large model, so as to correct the impact of the above method on the distribution of the results, so as to achieve the goal of keeping the distribution of the final output results unchanged.

[0041] Through the foregoing judgment process, the present invention ensures that the results obtained by this method and the results directly derived by large model reasoning are identically distributed.

[0042] Figure 2 An embodiment of the process of judging the prediction result of the small model by the above model is given in FIG. , which includes the following steps:

[0043] a) First, use the small model to generate a prediction tree. Using the above method, greedily select the best expandable node each time until a preset end condition is reached (for example, reaching the generation limit N).

[0044] b) Input the obtained prediction tree into the large model to obtain the actual sampling probability of each token.

[0045] c) According to the actual probability distribution output by the large model and the probability distribution obtained by the previous small model, verify whether each token should be received one by one. The specific verification method is: for the current sampling result x of the current token, the probability distribution p of the large model, and the probability distribution q of the small model, receive the current token x with a probability of min(1,p / q). If the result is not received, check whether the next sampling result y of the current token should be received. Due to the impact of the previous non-reception on the probability distribution, in order to ensure that the final generated probability distribution remains unchanged, it is necessary to calculate a residual probability distribution as the actual sampling probability distribution of the next time, so as to achieve the sampling probability distribution of the current token unchanged. The calculation method of this residual probability distribution is max(0,pq) (that is, Relu function), and then the result is normalized (normalization is norm function) to obtain a probability distribution p'.

[0046] The key points of the present invention are:

[0047] 1. The probability distribution generated by the small model is used as an estimate of the acceptance rate of the corresponding token. Experiments have shown that this estimate is reasonable and reliable.

[0048] 2. In the process of generating tree structure prediction results, the next operation is dynamically determined based on the generated results, while not violating the limitation of the method itself that the probability distribution of the final output results will not be changed.

[0049] 3. In the process of generating tree structure prediction results, the end condition can be dynamically determined according to the current situation.

[0050] The present invention directly uses the probability distribution generated by the small model as an estimate of the reception rate of the corresponding token. This probability distribution is obtained by simply processing the direct generation result of the small model. The probability distribution obtained by directly using the direct generation result of the small model through a simple neural network is essentially consistent with that obtained by directly using the processing, and similar effects can be obtained. Better performance can be obtained after tuning in specific scenarios.

[0051] The present invention dynamically determines the next operation according to the generated results in the process of generating tree structure prediction results, and formulates a dynamic decision-making strategy with the goal of optimizing the number of received tokens. When the optimization target is adjusted, the valuation function and decision-making method dynamically decided according to statistical information may also change accordingly.

[0052] Specific application scenarios of the present invention:

[0053] The method of the present invention can be applied to a variety of natural language processing tasks. For example, for the task of generating a summary from a given text, when llama2-13B is selected as the target model and llama2-68M is used as a small model for acceleration, each step of reasoning of the target model can receive an average of 4.59 characters, and a reasoning acceleration effect of 2.82 times is obtained. In addition, the present invention can ensure that the generation effect of the acceleration method is equivalent to that of directly generating using the target model.

[0054] Another embodiment of the present invention provides a system for accelerating large model reasoning using dynamic tree structure speculative decoding, which includes:

[0055] A small model prediction module is used to process the input text using a small model to generate predicted characters in a tree structure;

[0056] The large model inference module is used to input the input text and the predicted characters generated by the small model into the large model, obtain the output result of the large model through the large model inference process, and determine whether to accept the predicted characters generated by the small model, so as to obtain the final generated text.

[0057] The division of the above modules is only an example. In actual applications, the above functions can be assigned to different functional modules as needed to complete all or part of the functions described in the above method. The specific working process of each module can refer to the corresponding process in the above method embodiment, which will not be repeated here.

[0058] Another embodiment of the present invention provides a computer device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0059] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the steps of the method of the present invention are implemented.

[0060] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and implement it accordingly. It can be understood by those skilled in the art that various replacements, changes and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the contents disclosed in the embodiments of this specification, and the scope of protection of the present invention shall be subject to the scope defined in the claims.

Claims

1. A method for accelerating large model reasoning using dynamic tree structure speculative decoding, characterized in that: The following steps are involved: Use a small model to process the input text and generate predicted characters in a tree structure; The input text and the predicted characters generated by the small model are input into the large model together, the output result of the large model is obtained through the large model reasoning process, and it is determined whether to accept the predicted characters generated by the small model, so as to obtain the final generated text.

2. The method according to claim 1, characterized in that Before reaching the prediction end condition, the small model continuously predicts new characters to be added to the tree structure; the small model processes each expandable node of the currently generated tree structure, and obtains the probability distribution of each expandable node generating the next character through the standard small model reasoning process; the probability of each character output by the small model is directly used as an estimate of the probability that the character will eventually be accepted by the large model, that is, the acceptance rate.

3. The method according to claim 2, characterized in that The probability distribution obtained by the small model is: Among them, prob i is the probability distribution obtained; temperature is a non-negative number, generally an empirical value of 0 to 2; logit is the output of the small model, an n-dimensional floating-point vector, where n is the size of the vocabulary; according to prob i The node with the largest reception rate among all candidate nodes is selected and expanded to join the predicted tree structure.

4. The method according to claim 1, characterized in that: When the small model reaches the exit condition of the prediction tree structure, the generation of the current prediction tree structure is terminated; the exit condition is set as the size of the tree structure reaching a threshold, or is set as a threshold of the estimated reception rate of the predicted characters in the tree structure.

5. The method according to claim 1, characterized in that The optimal tree structure obtained by the small model is a tree structure with the maximum sum of the receiving rates of all nodes under a given number of nodes in the tree; The predicted tree structure is efficiently obtained using a priority queue algorithm, wherein the priority queue algorithm includes: adding all candidate nodes to the same priority queue to ensure that the node popped out each time is the node with the highest reception rate in the current queue; After each pop-up node, a new candidate node is updated and added to the priority queue; the new candidate node can be the extended node, including child nodes and sibling nodes. The premise for the child node to be received is that the current node is received, and the premise for the sibling node to be received is that the current node is not received.

6. The method according to claim 1, characterized in that The large model determines whether to accept the predicted characters generated by the small model, including: comparing the probability distribution of subsequent nodes obtained by the large model with the probability distribution of subsequent nodes predicted by the small model, determining whether the results generated by the small model match the results generated by the large model, and then deciding whether to accept the results of the small model.

7. The method according to claim 6, characterized in that The large model determines whether to receive the predicted character generated by the small model, including: Map each random event corresponding to the probability distribution of the small model and each random event corresponding to the probability distribution obtained by the large model in a fixed rule; After completing the mapping of random events, each random event of the small model will be in two states: mapped to the same event in the large model / mapped to different events in the large model; if the character sampled by the small model from the probability distribution belongs to the "mapped to the same event in the large model" state, it means that the large model has verified the character and the character will be accepted, otherwise the character will not be accepted.

8. A system for accelerating large model reasoning using dynamic tree structure speculative decoding, characterized in that: include: A small model prediction module is used to process the input text using a small model to generate predicted characters in a tree structure; The large model inference module is used to input the input text and the predicted characters generated by the small model into the large model, obtain the output result of the large model through the large model inference process, and determine whether to accept the predicted characters generated by the small model, so as to obtain the final generated text.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Images