Method and system for accelerating model reasoning by using dynamic tree structure speculative decoding
By adopting a dynamic tree structure in speculative decoding, dynamically generating and adjusting the prediction tree structure, the problems of inferred length limitation and low resource utilization in the existing technology are solved, and more efficient model inference and resource utilization are achieved.
Patent Information
- Application Number
- CN202411952635.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-04-25
AI Technical Summary
When the existing chain speculative decoding technology increases, the expected returns are decreasing exponentially, resulting in a short effective speculation length, which cannot significantly improve the model inference efficiency, and cannot make full use of hardware resources, resulting in waste of performance.
The dynamic tree structure speculative decoding method is adopted to dynamically generate the prediction tree structure through the prediction model. Each step is to adjust the subsequent prediction tree structure according to the current probability distribution, and dynamically expand the prediction tree to improve the inference effect.
It improves the model's inference effect in complex scenarios, significantly improves the model's inference efficiency, can make full use of hardware resources, and reduces performance waste.
Smart Images
Figure CN119884931A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for accelerating model reasoning by using dynamic tree structure speculative decoding. Background Art
[0002] At present, with the continuous development of language models, generative models based on autoregression have made significant progress in the field of natural language processing. These models are widely used in tasks such as text generation and machine translation by gradually generating each token in the output sequence. However, due to the high complexity of the model and the dependence of the reasoning process on the generation results of the previous step, the efficiency of generating sequences has become a bottleneck in practical applications.
[0003] In order to solve the efficiency problem in model reasoning, speculative decoding (SD) technology was proposed. This technology generates preliminary prediction results by using a lightweight guessing model and submits them to the target model for verification, thereby reducing the direct calculation amount of the target model. However, the existing chain speculative decoding technology has the following significant limitations:
[0004] Chained speculative decoding assumes that the probability of receiving each token in the generated sequence is independent, and the condition for the subsequent token to be accepted depends on the reception result of the previous token. Therefore, as the speculation length increases, the overall expected return decreases exponentially, resulting in a shorter actual effective speculation length and failing to significantly improve reasoning efficiency.
[0005] Most current models are deployed on general-purpose graphics processing units (GPGPUs), whose multi-core high-throughput architectures require tasks to be highly parallel. However, the short inference length of chain methods limits the ability of parallel processing, making it impossible to fully utilize hardware resources, resulting in performance waste.
[0006] To overcome the limitations of the chain method, researchers proposed a tree-structured speculative decoding technique that dynamically generates a prediction tree structure to expand the inference results. However, the existing tree-structured speculative decoding method still has the following problems:
[0007] Existing methods require obtaining the probability distribution of the model's generated results in advance and assume that the distribution remains unchanged throughout the decoding process. This assumption is seriously inconsistent with the actual generation scenario, because the model dynamically adjusts the generation probability according to the context during the generation process.
[0008] The current method fails to dynamically adjust the prediction results when generating the prediction tree, but generates the prediction tree based on a fixed inference scheme. This static method does not take into account the unexpected situations that may occur during the actual generation process, resulting in poor inference results in complex scenarios.
[0009] Therefore, a solution is urgently needed. Summary of the invention
[0010] One of the purposes of the present invention is to provide a method for accelerating model reasoning by speculative decoding using a dynamic tree structure. Using a prediction model, a prediction intermediate result, i.e., a partial prediction tree structure, is obtained at each step of the prediction process until the entire prediction tree structure is generated. A probability distribution is obtained at each step, and this probability distribution can reflect the reasoning process to a certain extent. The subsequent prediction tree structure is determined based on the probability distribution adjustment of this intermediate process, which is more in line with actual application scenarios, dynamically generates prediction trees, and improves the inference effect of the model in complex scenarios.
[0011] The method for accelerating model reasoning by using dynamic tree structure speculative decoding provided by an embodiment of the present invention includes:
[0012] S1. Based on the prediction model, dynamically generate the possible output results and the corresponding current probability distribution after the input data is input into the target model;
[0013] S2, based on the current probability distribution, estimate the probability that the possible output results will be received by the target model under the speculative decoding method;
[0014] S3, based on the probability that the possible output results are accepted by the target model, expand and generate the prediction tree structure;
[0015] S4, repeating steps S1 to S3 until the prediction tree structure is generated, and verifying the generation results in the generated prediction tree structure in parallel based on the target model;
[0016] S5. Output the verified generated result as the output result of the target model.
[0017] Optionally, the step S2, estimating the probability of a possible output result being received by a target model under a speculative decoding method based on a current probability distribution, includes:
[0018] The current probability distribution is used as the probability that the possible output results will be accepted by the target model under the speculative decoding method.
[0019] Optionally, the S3, based on the probability that the possible output result is received by the target model, expanding and generating the prediction tree structure, includes:
[0020] Based on the greedy strategy, the prediction tree structure is expanded and generated according to the probability that the possible output results are accepted by the target model;
[0021] and / or,
[0022] Based on the layer-by-layer generation strategy, the prediction tree structure is expanded and generated according to the probability that the possible output results are accepted by the target model.
[0023] Optionally, the greedy strategy is based on which the prediction tree structure is dynamically expanded and generated according to the probability that the possible output result is received by the target model, including:
[0024] Based on the possible output results, an initial prediction tree structure is constructed;
[0025] Predict the expected benefit of expanding each node in the initial prediction tree structure based on the probability that the possible output results are accepted by the target model;
[0026] Select the node with the largest expected benefit for expansion.
[0027] Optionally, predicting the expected benefit of expanding each node in the initial prediction tree structure based on the probability that the possible output result is received by the target model includes:
[0028] The formula for calculating the expected benefit of expanding each node in the initial prediction tree structure is as follows:
[0029]
[0030] where t u Represents the node currently performing expected return calculation, Yes u At the i-th ancestor node in the initial prediction tree structure, SD[Y] is the estimated probability of node Y being accepted, where Y is an arbitrary variable; is in the initial prediction tree structure u The jth sibling node before, i.e. t u The jth child node of the parent node, and the order of the child node is in t u Previously, ∏ represented the product operation.
[0031] Optionally, based on a layer-by-layer generation strategy, the prediction tree structure is dynamically expanded and generated according to the probability that the possible output results are received by the target model, including:
[0032] From each layer of the initial prediction tree structure, nodes with expected benefits greater than a threshold are selected for expansion.
[0033] Optionally, in S4, based on the target model, the generated results in the dynamically expanded prediction tree structure are verified in parallel, including:
[0034] Based on the mapping relationship between the sampling distribution generated by the prediction model and the sampling distribution generated by the target model, the generated results in the dynamically expanded prediction tree structure are verified in parallel.
[0035] Optionally, after outputting the verified generated result as the output result of the target model, it also includes:
[0036] The generated results that fail the verification are resampled and corrected.
[0037] Optionally, the sampling distribution generated by the prediction model is associated with the sampling distribution generated by the target model through a mapping relationship.
[0038] The system for accelerating model reasoning by using dynamic tree structure speculative decoding provided by an embodiment of the present invention is characterized by comprising:
[0039] A generation module is used to execute S1 and dynamically generate possible output results and corresponding current probability distribution after the input data is input into the target model based on the prediction model;
[0040] An estimation module, used to execute S2, based on the current probability distribution, estimate the probability that the possible output result is received by the target model under the speculative decoding method;
[0041] An extension module is used to execute S3 and expand the generated prediction tree structure based on the probability that the possible output results are accepted by the target model;
[0042] A verification module, used to execute S4, repeat steps S1 to S3, until the prediction tree structure is generated, and verify the generation results in the generated prediction tree structure in parallel based on the target model;
[0043] The output module is used to execute S5 and output the verified generation result as the output result of the target model.
[0044] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.
[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0047] Figure 1 A schematic diagram of a method for accelerating model reasoning by using dynamic tree structure speculative decoding in an embodiment of the present invention;
[0048] Figure 2 This is a result diagram of the test model reasoning acceleration effect in an embodiment of the present invention;
[0049] Figure 3 This is another result diagram of the test model reasoning acceleration effect in an embodiment of the present invention;
[0050] Figure 4 This is another result diagram of the test model reasoning acceleration effect in an embodiment of the present invention;
[0051] Figure 5 A schematic diagram of a process for dynamically expanding and generating adjacent steps of a prediction tree structure in an embodiment of the present invention;
[0052] Figure 6 It is another process schematic diagram of generating adjacent steps for dynamically expanding the prediction tree structure in an embodiment of the present invention;
[0053] Figure 7 A schematic diagram of a mapping relationship between a sampling distribution generated by a prediction model and a sampling distribution generated by a target model in an embodiment of the present invention;
[0054] Figure 8 It is a schematic diagram of a system for accelerating model reasoning by using dynamic tree structure speculative decoding in an embodiment of the present invention. DETAILED DESCRIPTION
[0055] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0056] The embodiment of the present invention provides a method for accelerating model reasoning by using dynamic tree structure speculative decoding, such as Figure 1 As shown, including:
[0057] S1. Based on the prediction model, dynamically generate the possible output results and the corresponding current probability distribution after the input data is input into the target model;
[0058] S2, based on the current probability distribution, estimate the probability that the possible output results will be received by the target model under the speculative decoding method;
[0059] S3, based on the probability that the possible output results are accepted by the target model, expand and generate the prediction tree structure;
[0060] S4, repeating steps S1 to S3 until the prediction tree structure is generated, and verifying the generation results in the generated prediction tree structure in parallel based on the target model;
[0061] The generated results that pass the verification are output as the output results of the target model.
[0062] In the above technical solution, both the prediction model and the target model are generative language models; the prediction model is a model with much smaller parameters and inference delay than the target model, and it is expected that the prediction model can achieve similar capabilities or behaviors as the target model in multiple tasks (for example, similar capabilities in simple tasks, the same vocabulary, etc.). The prediction model has fewer parameters than the target model, so the inference speed is faster, but it does not perform as well as the target model in some difficult tasks; generally, a model with fewer parameters in the same series / framework is selected as the prediction model. Input data refers to the data to be input into the target model to obtain a response result. After the input data is input into the prediction model, the prediction model will generate a probability distribution at each step during the generation process, corresponding to the probability of each token being selected in this step. The prediction tree structure is a multi-branch tree structure in logic.
[0063] This application uses a prediction model, and a prediction intermediate result, that is, a partial prediction tree structure, will be obtained at each step of the prediction process until the entire prediction tree structure is generated. A probability distribution will be obtained at each step, and this probability distribution can reflect the reasoning process to a certain extent. The subsequent prediction tree structure is determined based on the probability distribution adjustment of this intermediate process, which is more in line with actual application scenarios, dynamically generates prediction trees, and improves the inference effect of the model in complex scenarios.
[0064] like Figures 2 to 5 As shown in the figure, prediction models / target models with different parameter scales are selected to verify the acceleration effect of the method. The selected datasets are: wikipedia, cnndailymain, openwebtext, GSM8k, and MT-bench. The selected prediction models / target models are: JF68M / Llama2-7B, JF68M / Llama2-13B, Llama2-7B / Llama2-70B. The selected comparison methods are Sequoia and Specinfer. Among them, Wikipedia is a Wikipedia dataset that contains a large number of articles on the Wikipedia website, covering a wide range of topics, and is often used to train language models and perform natural language processing tasks. CNN / DailyMail is a news summary dataset that contains news articles from CNN and the Daily Mail and their corresponding summaries, and is often used for text summarization and reading comprehension tasks. OpenWebText is an open source text dataset that imitates OpenAI's WebText dataset and contains diverse texts captured from the Internet for training language models. GSM8k is a math problem solving dataset that contains 8.5K elementary school-level math problems. It is used to test and train the model’s ability to solve math problems.
[0065] MT-Benc is a multi-task benchmark test dataset used to evaluate the performance of models on multiple tasks, covering multiple fields such as question answering, reasoning, and generation. These datasets are datasets of different fields and tasks commonly used in language model research. Because the present invention is a general model acceleration technology, common datasets of different research tasks are selected for experiments.
[0066] Figure 2 The average latency of generating a single token (the number of generated tokens in a single inference is in brackets), the prediction model is JF68m, the target model is Llama2-13B, and the size of each prediction tree is 64;
[0067] Figure 3 The average latency of generating a single token (the number of generated tokens in a single inference is in brackets), the prediction model is Llama2-7B, the target model is Llama2-70B, and the tree size of each prediction is 64;
[0068] Figure 4 , the average latency to generate a single token (the number of generated tokens in a single inference is in brackets), the prediction model is Llama2-7B, and the target model is Llama2-70B. The tree size for each prediction is 64.
[0069] The test results show that the method tested on different prediction model / target model combinations, different data sets, and different model temperatures was compared with the best existing related methods. The experimentally proven method has a higher reasoning acceleration effect and the average number of generated tokens per target model generation than existing related work.
[0070] In one embodiment, the step S2, estimating the probability of a possible output result being received by a target model under a speculative decoding method based on a current probability distribution, comprises:
[0071] The current probability distribution is used as the probability that the possible output results will be accepted by the target model under the speculative decoding method.
[0072] In the above technical solution, through experimental statistics, it is found that the probability of each token obtained by the general prediction model at a certain step is proportional to the probability of this token being verified by the target model during the verification process (i.e., the acceptance rate); therefore, the probability distribution at the current position is used as the estimated acceptance rate after the token is sampled. Each step of generation is obtained through probability sampling, and the expected acceptance rate is estimated for the sampling result, not for this generation, that is, for different sampling results, the estimate of the expected acceptance rate of this generation is dynamic. Secondly, the probability distribution used for each step of generation is also different, and is not determined only by the position of this step of generation in the tree structure. In the implementation, the probability of this token predicted by the prediction model, that is, the current probability distribution, is directly used as the estimated acceptance rate.
[0073] In one embodiment, the step S3, based on the probability that the possible output result is received by the target model, expanding and generating the prediction tree structure, comprises:
[0074] Based on the greedy strategy, the prediction tree structure is expanded and generated according to the probability that the possible output results are accepted by the target model;
[0075] and / or,
[0076] Based on the layer-by-layer generation strategy, the prediction tree structure is expanded and generated according to the probability that the possible output results are accepted by the target model.
[0077] In one embodiment, the method of dynamically expanding and generating a prediction tree structure based on a greedy strategy according to the probability that a possible output result is received by a target model includes:
[0078] Based on the possible output results, an initial prediction tree structure is constructed;
[0079] Predict the expected benefit of expanding each node in the initial prediction tree structure based on the probability that the possible output results are accepted by the target model;
[0080] Select the node with the largest expected benefit for expansion.
[0081] In the above technical solution, the prediction tree initially uses the input content as the root node, and directly inputs the input content into the prediction model, so as to obtain the prediction candidate set generated by the prediction model in the first step and the corresponding probability distribution (that is, the probability of each token generated by a probability distribution corresponding to sampling); initially, the above state is used as a partial prediction tree, and the prediction tree is cyclically expanded until the preset termination condition is reached (the number of tree nodes reaches the upper limit), and the partial prediction tree of each step of expansion uses the result obtained in the previous step (the prediction tree structure generated by the previous step expansion) as the input, that is, the initial prediction tree structure is constructed.
[0082] like Figure 5 and Figure 6As shown in , these are two adjacent steps in the process of repeatedly constructing a prediction tree. Each step is annotated with the estimated acceptance rate of each node obtained during the construction process, as well as the actual expected contribution of each node obtained from the acceptance rate of a single node based on the dependency of the tree structure; for example, Figure 5 In the example, two results B and E are predicted after input A. The expected acceptance rate of B is 0.7, and the expected contribution is 0.7. The expected acceptance rate of E is 0.9, but E can only be accepted after B is accepted, so the actual contribution of E is 0.7*0.9=0.63. These two are the highest among all the results.
[0083] Figure 5 In the example, the potential expandable node is that if B is rejected, the next prediction may be made at this position, and the result is C. If E is rejected, the next prediction may be made at this position, and the result is F. Through calculation, it can be obtained that the expected contribution of C is 0.15 and the expected contribution of F is 0.09, so the point C with the highest expected return is selected and added to the prediction tree.
[0084] In the next step, some prediction results include ABCE, and the expandable nodes include the next prediction result F after B is rejected, the next prediction result D after C is rejected, and the successor node G of C when C is accepted. The expected return of G of 0.12 calculated in DFG is optimal, so G is added to the prediction tree structure in this step.
[0085] In one embodiment, the predicting the expected benefit of expanding each node in the initial prediction tree structure based on the probability that the possible output result is received by the target model includes:
[0086] The formula for calculating the expected benefit of expanding each node in the initial prediction tree structure is as follows:
[0087]
[0088] where t u Represents the node currently performing expected return calculation, Yes u At the ith ancestor node in the initial prediction tree structure, SD[x] is the estimated probability of node x being accepted, where x is an arbitrary variable, such as t u ; is in the initial prediction tree structure u The jth sibling node before, i.e. t u The jth child node of the parent node, and the order of the child node is in t u Previously, ∏ represented the product operation. For any node x on the tree, the probability of its final acceptance consists of two parts:
[0089] Event a: the parent node of the current node is received (that is, all previous predictions of the prediction sequence where the sample is located this time are received);
[0090] Event b, all sibling nodes before the current node are rejected (that is, all previous samples at the location of this sampling are rejected; assuming that this sampling is the k-th sampling of this position pos, the k-th sampling will be performed only if the previous k-1 samples are rejected);
[0091] The event a can be expressed as:
[0092]
[0093] Event b can be expressed as:
[0094]
[0095] Multiplying this node's own receiving rate by the node, when selecting this node, the expected contribution of this node can be expressed as:
[0096]
[0097] With the expected contribution of each node, the optimal tree structure can be constructed greedily.
[0098] In one embodiment, based on a layer-by-layer generation strategy, according to the probability that a possible output result is received by a target model, a prediction tree structure is dynamically expanded and generated, including:
[0099] From each layer of the initial prediction tree structure, nodes with expected benefits greater than a threshold are selected for expansion.
[0100] In the above technical solution, the aforementioned method can only select the optimal nodes one by one. In this way, the prediction model needs to be called n times when predicting n nodes, which offsets the acceleration effect of the method itself to a certain extent. Note that for any tree structure, under the autoregressive generation paradigm of the large language model, at least the number of layers needs to be called. Therefore, the goal is to enable the prediction tree to be generated layer by layer. Note that the estimated expected return is monotonically decreasing, so you only need to select a suitable threshold and then sample all nodes with expected returns greater than this threshold to obtain an optimal tree structure. The expected return of the node can be obtained layer by layer. In this way, a method for generating prediction trees layer by layer is obtained.
[0101] In one embodiment, in S4, based on the target model, the generated results in the dynamically expanded prediction tree structure are verified in parallel, including:
[0102] Based on the mapping relationship between the sampling distribution generated by the prediction model and the sampling distribution generated by the target model, the generated results in the dynamically expanded prediction tree structure are verified in parallel.
[0103] In the above technical solution, the process of generating results by the prediction model and the process of directly generating results by the target model are both achieved through sampling. If the verification is only based on whether the results are equal, the acceptance rate will be lost due to sampling. By mapping the two, the acceptance rate can be improved while ensuring that the probability distribution remains unchanged. Figure 7 As shown, the probability of the prediction model generating A, B, and C is 0.3, 0.2, and 0.5 respectively, and the probability of the target model generating A, B, and C is 0.2, 0.5, and 0.3 respectively. If the result is judged by whether it is consistent, the expected acceptance rate is 0.3*0.2+0.2*0.5+0.5*0.3=0.31.
[0104] The method of constructing the mapping is that if the probability of the target model on A is less than the probability of the prediction model on A, then the mapping is required to satisfy the target model. All the probabilities of A are mapped to the prediction model's A, and the remaining probability of the prediction model on A can be mapped to other events (results) of the target model. On the contrary, if the probability of B on the prediction model is small, the mapping requires that all B of the prediction model is mapped to B of the target model, and the remaining B of the target model can be mapped arbitrarily. The optimal mapping constructed by following the above two principles can achieve the optimal expected acceptance rate. As long as the results of the prediction model and the results of the target model correspond, it is considered to have passed the verification. In the above example, the expected acceptance rate is 0.2+0.2+0.3=0.7, which is significantly higher than the direct matching of 0.31.
[0105] In one embodiment, after outputting the verified generated result as the output result of the target model, the method further includes:
[0106] The generated results that fail the verification are resampled and corrected.
[0107] In the above technical solution, for node x, the target model will be The probability of receiving this result; where T[x] is the probability of the target model directly generating x, D[x] is the probability of the prediction model generating x, and min means taking the minimum value.
[0108] When the target model is verified, if the first predicted result is not received at the same position, assuming that the generated tree structure also generates a second predicted result here, then a second verification can be performed. Since the prediction model uses the method of sampling without replacement when predicting, the probability of generating the second result will change to a certain extent: the result of the previous sampling will be set to 0, and then the remaining part will be normalized to obtain a new probability distribution D'. This probability D' distribution will be directly recorded, corresponding to the probability distribution of the prediction model on the second prediction.
[0109] Since a mapping is constructed between the probability distribution of the prediction model output and the probability distribution of the target model output, the rejection of the prediction results only corresponds to those cases that do not match. Therefore, the probability distribution of the target model after rejection is also called adjustment, subtracting those cases that match. The method of subtracting the matching cases is to calculate the expression
[0110] T' = norm(relu(TD))
[0111] Where T is the probability distribution of the target model output, D is the probability distribution of the prediction model output, relu(TD) sets the part of the TD result that is less than 0 to 0, and norm() normalizes these results to get a new probability distribution. T' is the probability distribution of the corrected target model. Sampling directly from this probability distribution can get the result directly generated by the target model.
[0112] When the first prediction result is rejected, the second prediction result is verified. At this time, for the second prediction result, the probability distribution of the prediction model is the probability distribution D' recorded directly above, and the probability distribution of the target model is T'.
[0113] The embodiment of the present invention provides a system for accelerating model reasoning by using dynamic tree structure speculative decoding, such as Figure 8 As shown, including:
[0114] Generation module 1, used to execute S1, dynamically generate possible output results and corresponding current probability distribution after input data is input into the target model based on the prediction model;
[0115] An estimation module 2 is used to execute S2, estimating the probability of a possible output result being received by a target model under a speculative decoding method based on a current probability distribution;
[0116] Extension module 3, used to execute S3, and expand and generate a prediction tree structure based on the probability that the possible output result is received by the target model;
[0117] Verification module 4, used to execute S4, repeat steps S1 to S3, until the prediction tree structure is generated, and verify the generation results in the generated prediction tree structure in parallel based on the target model;
[0118] The output module 5 is used to execute S5 and output the verified generation result as the output result of the target model.
[0119] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A method for accelerating model reasoning by using dynamic tree structure speculative decoding, characterized in that: include: S1. Based on the prediction model, dynamically generate the possible output results and the corresponding current probability distribution after the input data is input into the target model; S2, based on the current probability distribution, estimate the probability that the possible output results will be received by the target model under the speculative decoding method; S3, based on the probability that the possible output results are accepted by the target model, expand and generate the prediction tree structure; S4, repeating steps S1 to S3 until the prediction tree structure is generated, and verifying the generation results in the generated prediction tree structure in parallel based on the target model; S5. Output the verified generated result as the output result of the target model.
2. The method for accelerating model reasoning by dynamic tree structure speculative decoding as claimed in claim 1, characterized in that: S2, based on the current probability distribution, estimating the probability that the possible output result is received by the target model under the speculative decoding method, includes: The current probability distribution is used as the probability that the possible output results will be accepted by the target model under the speculative decoding method.
3. The method for accelerating model reasoning by dynamic tree structure speculative decoding as claimed in claim 1, characterized in that: S3, based on the probability that the possible output result is received by the target model, expanding and generating a prediction tree structure, comprises: Based on the greedy strategy, the prediction tree structure is expanded and generated according to the probability that the possible output results are accepted by the target model; and / or, Based on the layer-by-layer generation strategy, the prediction tree structure is expanded and generated according to the probability that the possible output results are accepted by the target model.
4. The method for accelerating model reasoning by dynamic tree structure speculative decoding as claimed in claim 3, characterized in that: The greedy strategy is based on the probability that the possible output results are received by the target model, and the prediction tree structure is dynamically expanded and generated, including: Based on the possible output results, an initial prediction tree structure is constructed; Predict the expected benefit of expanding each node in the initial prediction tree structure based on the probability that the possible output results are accepted by the target model; Select the node with the largest expected benefit for expansion.
5. The method for accelerating model reasoning by dynamic tree structure speculative decoding as claimed in claim 4, characterized in that: The method of predicting the expected benefit of expanding each node in the initial prediction tree structure based on the probability of the possible output result being received by the target model includes: The formula for calculating the expected benefit of expanding each node in the initial prediction tree structure is as follows: where t u Represents the node currently performing expected return calculation, Yes u At the i-th ancestor node in the initial prediction tree structure, SD[Y] is the estimated probability of node Y being accepted, where Y is an arbitrary variable; is in the initial prediction tree structure u The jth sibling node before, i.e. t u The jth child node of the parent node, and the order of the child node is in t u Previously, ∏ represented the product operation.
6. The method for accelerating model reasoning by dynamic tree structure speculative decoding as claimed in claim 4, characterized in that: Based on the layer-by-layer generation strategy, the prediction tree structure is dynamically expanded and generated according to the probability that the possible output results are accepted by the target model, including: From each layer of the initial prediction tree structure, nodes with expected benefits greater than a threshold are selected for expansion.
7. The method for accelerating model reasoning by dynamic tree structure speculative decoding as claimed in claim 1, characterized in that: In S4, based on the target model, the generated results in the dynamically expanded prediction tree structure are verified in parallel, including: Based on the mapping relationship between the sampling distribution generated by the prediction model and the sampling distribution generated by the target model, the generated results in the dynamically expanded prediction tree structure are verified in parallel.
8. The method for accelerating model reasoning by dynamic tree structure speculative decoding as claimed in claim 1, characterized in that: After the verified generated results are output as the output results of the target model, it also includes: The generated results that fail the verification are resampled and corrected.
9. A system for accelerating model reasoning using dynamic tree structure speculative decoding, characterized in that: include: A generation module is used to execute S1 and dynamically generate possible output results and corresponding current probability distribution after the input data is input into the target model based on the prediction model; An estimation module, used to execute S2, based on the current probability distribution, estimate the probability that the possible output result is received by the target model under the speculative decoding method; An extension module is used to execute S3 and expand the generated prediction tree structure based on the probability that the possible output results are accepted by the target model; A verification module, used to execute S4, repeat steps S1 to S3, until the prediction tree structure is generated, and verify the generation results in the generated prediction tree structure in parallel based on the target model; The output module is used to execute S5 and output the verified generation result as the output result of the target model.