Tree speculation sampling method and device for generative model reasoning

Abstract

The invention provides a tree speculation sampling method and device for generative model reasoning. The method comprises the following steps: inputting an input sequence into a generative model; the generative model performs reasoning calculation based on the input sequence to generate a plurality of candidate tokens of a tree structure; receiving a part of candidate tokens in the tree structure; updating the input sequence through the partial candidate tokens to generate a current sequence; performing next reasoning calculation based on the current sequence, and recalculating the tokens which are not cached in the next reasoning calculation; and when a termination condition is met, outputting the generated final sequence. According to the tree speculation sampling method and device for generative model reasoning, various possible generation paths can be explored in parallel, the efficiency bottleneck caused by gradual reasoning is avoided, calculation resources are saved in the reasoning calculation process, and the balance between caching efficiency and flexibility is achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer information processing, and in particular to a tree-like speculative sampling method and device for generative model reasoning. Background Art

[0002] In recent years, generative models have been widely used in fields such as natural language processing (NLP), computer vision, and speech generation. These models can generate new data that conforms to the target distribution based on existing data, such as continuing text, generating images, or synthesizing speech. However, as the scale of generative models continues to expand (such as language models with billions or even hundreds of billions of parameters), the complexity of their reasoning calculations has increased significantly, resulting in limited generation speed and excessive resource consumption. Therefore, optimizing the reasoning efficiency of generative models has become an important research direction.

[0003] Speculative sampling is an optimization method designed to improve the efficiency of generative model reasoning. The core idea is to generate and verify multiple candidate results in advance by predicting the future output of the model. Speculative sampling can reduce the time overhead of step-by-step generation while improving the parallelism of reasoning. Tree-like speculative sampling constructs a tree structure that allows the model to generate multiple candidate paths for parallel exploration at each time step.

[0004] Although speculative sampling and tree-based speculative sampling have significant advantages in improving the efficiency of generative model reasoning, in actual implementation, they still face the following difficulties in computing speed and cache resource usage.

[0005] Therefore, a new tree-structured speculative sampling method and device for generative model reasoning is needed.

[0006] The above information disclosed in the Background section is only for enhancement of understanding of the background of the present application and therefore it may contain information that does not constitute the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0007] In view of this, the present application provides a tree-like speculative sampling method and device for generative model reasoning, which combines partial acceptance + recalculation, DFS optimized traversal and other mechanisms to solve the following problems in the prior art: Insufficient reasoning efficiency: The generation speed is significantly improved through multi-path parallel exploration of the tree structure.

[0008] Low cache utilization: Use partial acceptance and dynamic recalculation mechanisms to optimize the storage and computing efficiency of the kv cache.

[0009] Insufficient path optimization: Arrange the path in order through DFS traversal, increase the continuous cache length, and reduce the model calculation overhead.

[0010] This application is suitable for tasks that require the generation of complex sequences, such as natural language processing, speech generation, and image description, and has significant technical advantages and broad application prospects.

[0011] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0012] According to one aspect of the present application, a tree-like speculative sampling method for generative model reasoning is proposed, the method comprising: inputting an input sequence into a generative model; the generative model performs reasoning calculation based on the input sequence to generate multiple candidate tokens in a tree structure; accepts some candidate tokens in the tree structure; updates the input sequence through the some candidate tokens to generate a current sequence; performs the next reasoning calculation based on the current sequence, and recalculates the uncached tokens in the next reasoning calculation; and outputs the generated final sequence when a termination condition is met.

[0013] In an exemplary embodiment of the present application, inputting an input sequence into a generative model includes: arranging the input sequence in an order of DFS traversal and inputting the sequence into the generative model.

[0014] In an exemplary embodiment of the present application, the input sequence is arranged in the order of DFS traversal and input into the generative model, including: inputting the input sequence into a draft model to generate multiple candidate tokens; caching the kv cache information corresponding to the multiple candidate tokens; arranging the multiple candidate tokens in the order of DFS traversal and inputting them into the generative model.

[0015] In an exemplary embodiment of the present application, the original sequence is input into the draft model to generate multiple candidate tokens, including: inputting the original sequence into the draft model multiple times, and generating the multiple candidate tokens through multiple speculative sampling processes.

[0016] In an exemplary embodiment of the present application, the generative model performs reasoning calculation based on the input sequence to generate multiple candidate tokens in a tree structure, and further includes: caching kv cache information corresponding to the multiple candidate tokens.

[0017] In an exemplary embodiment of the present application, accepting some candidate tokens in the tree structure includes: accepting k candidate tokens in the tree structure and caching kv cache information of p tokens; discarding kv cache information of kp tokens that are not cached in the some candidate tokens.

[0018] In an exemplary embodiment of the present application, accepting k candidate tokens in the tree structure and caching kv cache information of p tokens includes: accepting k candidate tokens in the tree structure; caching kv caches of the first consecutive p tokens among the k candidate tokens.

[0019] In an exemplary embodiment of the present application, the input sequence is updated by the partial candidate tokens to generate the current sequence, including: inputting the partial candidate tokens into the medusa speculation model to generate n candidate tokens; adding kp uncached tokens and a new token to the input sequence; adding n candidate tokens to the input sequence to generate the current sequence.

[0020] In an exemplary embodiment of the present application, updating the input sequence by using the partial candidate tokens to generate the current sequence includes: adding kp tokens that are not cached and a new token to the input sequence to generate the current sequence.

[0021] In an exemplary embodiment of the present application, recalculating the uncached tokens in the next reasoning calculation includes: calculating the uncached kp tokens again for the next reasoning calculation.

[0022] According to one aspect of the present application, a tree-like speculative sampling device for generative model reasoning is proposed, and the device includes: an input module, used to input an input sequence into a generative model; an inference module, used for the generative model to perform inference calculation based on the input sequence to generate multiple candidate tokens in a tree structure; an acceptance module, used to accept some candidate tokens in the tree structure; an update module, used to update the input sequence through the some candidate tokens to generate a current sequence; a recalculation module, used to perform the next inference calculation based on the current sequence, and recalculate the uncached tokens in the next inference calculation; an output module, used to output the generated final sequence when the termination condition is met.

[0023] According to one aspect of the present application, an electronic device is proposed, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0024] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0025] According to the tree-like speculative sampling method and device for generative model reasoning of the present application, an input sequence is input into a generative model; the generative model performs reasoning calculation based on the input sequence to generate multiple candidate tokens in a tree structure; some candidate tokens in the tree structure are accepted; the input sequence is updated by the some candidate tokens to generate a current sequence; the next reasoning calculation is performed based on the current sequence, and the uncached tokens are recalculated in the next reasoning calculation; when the termination condition is met, the generated final sequence is output in a manner that can explore multiple possible generation paths in parallel, avoid the efficiency bottleneck caused by step-by-step reasoning, save computing resources in the process of reasoning calculation, and achieve a balance between cache efficiency and flexibility.

[0026] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other objects, features and advantages of the present application will become more apparent by describing in detail the exemplary embodiments thereof with reference to the accompanying drawings. The accompanying drawings described below are only some embodiments of the present application, and it is clear to a person skilled in the art that other accompanying drawings can be obtained from these accompanying drawings without creative effort.

[0028] Figure 1 The present invention is a flowchart of a tree-structured speculative sampling method for generative model reasoning according to an exemplary embodiment.

[0029] Figure 2 The present invention is a flowchart of a tree-structured speculative sampling method for generative model reasoning according to an exemplary embodiment.

[0030] Figure 3 is a schematic diagram of a tree-like speculative sampling method for generative model reasoning according to another exemplary embodiment.

[0031] Figure 4 The present invention is a flowchart of a tree-structured speculative sampling method for generative model reasoning according to another exemplary embodiment.

[0032] Figure 5 It is a block diagram of a tree-like speculative sampling device for generative model reasoning according to an exemplary embodiment.

[0033] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment.

[0034] Figure 7It is a block diagram of a computer-readable medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0037] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0039] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0040] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0041] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.

[0042] The technical abbreviations involved in this application are explained as follows: KV Cache (Key-Value Cache) is a cache mechanism in generative models that is used to store the keys and values ​​generated by the model in self-attention calculations. By caching this information, it is possible to avoid repeatedly calculating the attention matrix of the previous token each time a token is generated, thereby speeding up the reasoning process. This is a key technology to improve generation efficiency.

[0043] Token is the smallest unit for generative models to process text data. It may be a character, a word, or a subword. For example, in the sentence "I love programming", possible tokens are "I", "love", and "programming". Token is the basic unit of model input and output sequence.

[0044] DFS (Depth First Search) is a path traversal method that preferentially traverses along a branch to the deepest point in a tree structure, and then returns to the upper node to find other branches.

[0045] BFS (Breadth First Search) is another path traversal method. In the tree structure, it first traverses all nodes in the current layer, and then goes deep into the next layer.

[0046] Speculative Sampling is a method for optimizing generative model reasoning. It reduces the time of step-by-step generation and improves generation efficiency by predicting multiple candidate tokens in advance.

[0047] Tree-based Speculative Sampling is an optimization method based on speculative sampling that supports multi-path parallel exploration by building a tree structure. Each node represents a token, each branch represents a different candidate path, and the depth of the tree corresponds to the length of the generated sequence.

[0048] Draft Model is a small auxiliary model used to quickly generate candidate token sequences (draft tokens). The role of the draft model is to reduce the computational burden of the target model through fast prediction and improve the overall reasoning speed.

[0049] Mask is used to mask irrelevant information in generative models, such as masking future ungenerated tokens in self-attention mechanisms. In tree-like speculative sampling, masks are generated dynamically to ensure that only valid paths are processed for each token.

[0050] Figure 1 The flowchart of a tree-structured speculative sampling method for generative model reasoning according to an exemplary embodiment is shown. The tree-structured speculative sampling method 10 for generative model reasoning at least includes steps S101 to S112.

[0051] like Figure 1 As shown, in S101, the input sequence is input into the generative model. For example, the input sequence can be arranged in the order of DFS traversal and input into the generative model.

[0052] In one embodiment, the input sequence is arranged in the order of DFS traversal and input into the generative model, including: inputting the input sequence into the draft model to generate multiple candidate tokens; caching the kv cache information corresponding to the multiple candidate tokens; arranging the multiple candidate tokens in the order of DFS traversal and inputting into the generative model. More specifically, the original sequence can be input into the draft model multiple times, and the multiple candidate tokens can be generated through multiple speculative sampling processes.

[0053] More specifically, the input sequence can be arranged in the order of depth-first search (DFS) traversal and then input into the generative model. DFS traversal can maximize the cache length of consecutive tokens, thereby reducing the occurrence of cache interruptions and improving the utilization of kvcache.

[0054] Specifically, the input sequence can be first input into the draft model, and the draft model is used as an auxiliary generation model to quickly generate multiple candidate tokens. The kv cache information corresponding to the multiple candidate tokens generated by the draft model is cached so that it can be directly called in subsequent reasoning steps to avoid repeated calculations. Afterwards, the generated candidate tokens can be arranged in the order of DFS traversal and input into the target generative model. In this way, the generated path coherence and reasoning efficiency can be effectively improved.

[0055] More specifically, in another embodiment, the original sequence can be input into the draft model multiple times, and multiple candidate tokens can be generated through multiple speculative sampling processes. This method can increase the diversity of candidate paths and ensure that the coverage of the generated results is wider.

[0056] In S104, the generative model performs inference calculations based on the input sequence to generate multiple candidate tokens in a tree structure. The generative model performs inference calculations based on the input sequence to generate multiple candidate tokens in a tree structure. The characteristic of the tree structure is that each node represents a generated token, each branch represents a different candidate path, and the depth of the tree corresponds to the length of the generated sequence. This structure can support multi-path parallel exploration, thereby improving the generation efficiency.

[0057] In one embodiment, it further includes: caching the kv cache information corresponding to the multiple candidate tokens.

[0058] In S106, some candidate tokens in the tree structure are accepted. Some candidate tokens in the tree structure are selected and accepted. For example, k candidate tokens in the tree structure are accepted, and the kv cache information of p tokens is cached; the kv cache information of the k - p tokens that are not cached among the partial candidate tokens is discarded. More specifically, k candidate tokens in the tree structure are accepted; the kv cache of the first consecutive p tokens among the k candidate tokens is cached.

[0059] In this application, the mechanism of only caching the kv cache of partial tokens can effectively balance the cache efficiency and the computational burden, ensuring more reasonable resource allocation.

[0060] In a specific embodiment, in the inference service, first, the input sequence needs to be input into the generative model. If there is no KV Cache, each inference requires recalculation, and at this time, only the position of the mask needs to be calculated correctly. However, when there is a KV Cache, it needs to be effectively managed.

[0061] For example, the generative model accepts a part of the candidate tokens, such as: [You, eat, have, him, me, apple, pear, of, of, have, apple, pear, apple, pear, have] Assume that the sequence "You eat have" accepts the three draft tokens "him", "of", "pear", and an additional full stop "." is generated. The final cache update is as follows: [You, eat, have, him, of, pear,.] In this process, because the sequence "You eat have" itself is connected to the draft token "him", the KV Cache of "You eat have him" is retained, while "of pear." is dynamically added to the sequence as an additional part but not cached.

[0062] The advantage of this mechanism is that if there are few draft tokens, the calculation itself is in an unsaturated state, and adding one or two tokens to each sequence will only make the calculation more saturated, and will not significantly increase the calculation time. (As mentioned earlier, the time to calculate 4 tokens is only 20% more than the time to calculate 2 tokens.) If there are many draft tokens, for example, 12 draft tokens, even if the calculation is already saturated, calculating 13 tokens increases the number of tokens by 2 to 15 tokens, and the calculation time will only increase by 2 / 13=15.4% at most.

[0063] The time increase above is only the time to calculate the matrix multiplication of kv, which is about the same as the single memory copy, or even faster. There is a fixed delay time when calling the GPU separately.

[0064] In S108, the input sequence is updated by the partial candidate tokens to generate a current sequence. The input sequence is updated by the accepted partial candidate tokens to generate a current sequence.

[0065] In one embodiment, the input sequence is updated by using the partial candidate tokens to generate the current sequence, including: inputting the partial candidate tokens into the medusa speculation model to generate n candidate tokens; adding kp uncached tokens and a new token to the input sequence; adding n candidate tokens to the input sequence to generate the current sequence.

[0066] For the application of the Medusa model, the update step may include the following: Input the accepted partial candidate tokens into the medusa speculative model to generate n new candidate tokens. Add the uncached kp tokens and a newly generated token (such as a completed word) to the input sequence. Add the n new candidate tokens to the input sequence together to finally generate the current sequence.

[0067] In another embodiment, the input sequence is updated by the partial candidate tokens to generate the current sequence, including: adding kp uncached tokens and a new token to the input sequence to generate the current sequence. When the Medusa model is not introduced, the update can be completed by directly adding the uncached kp tokens and a newly generated token to the input sequence. This update method ensures the continuity of the input sequence, while allowing new tokens to flexibly supplement the generation path.

[0068] In S110, the next inference calculation is performed based on the current sequence, and the uncached tokens are recalculated in the next inference calculation; more specifically, the uncached kp tokens may be calculated again for the next inference calculation.

[0069] For kp tokens that are not cached, recalculate their kv cache information and add it to the inference calculation. This dynamic recalculation mechanism can ensure the correctness of the generated sequence while effectively utilizing cache resources and avoiding storing redundant information.

[0070] In S112, when the termination condition is met, the generated final sequence is output. The termination condition may include but is not limited to the following: Generation is complete: the generated sequence reaches the target length or a termination marker (such as ".") appears.

[0071] According to the tree-like speculative sampling method for generative model reasoning of the present application, an input sequence is input into a generative model; the generative model performs reasoning calculation based on the input sequence to generate multiple candidate tokens in a tree structure; some candidate tokens in the tree structure are accepted; the input sequence is updated by the some candidate tokens to generate a current sequence; the next reasoning calculation is performed based on the current sequence, and the uncached tokens are recalculated in the next reasoning calculation; when the termination condition is met, the generated final sequence is output in a manner that can explore multiple possible generation paths in parallel, avoid the efficiency bottleneck caused by step-by-step reasoning, save computing resources in the process of reasoning calculation, and achieve a balance between cache efficiency and flexibility.

[0072] The tree-like speculative sampling method for generative model reasoning in this application realizes an efficient generative model reasoning process by combining DFS optimized path traversal, draft model rapid generation of candidate tokens, dynamic kv cache management and other technologies. Compared with the traditional step-by-step reasoning method, this application can better utilize computing resources, improve the generation speed, and ensure the consistency and accuracy of the generated results.

[0073] It should be clearly understood that the present application describes how to form and use specific examples, but the principles of the present application are not limited to any details of these examples. On the contrary, based on the teaching of the content disclosed in the present application, these principles can be applied to many other embodiments.

[0074] The Draft Model and the Medusa speculation method both play important roles in tree-structured speculative sampling. The main differences between the two are as follows: The Draft Model has an independent KV Cache, while Medusa directly relies on the multiple prediction heads of the target model. At the same time, the common point between the two is that before the target model inference, the order of candidate tokens needs to be converted from BFS (breadth-first search) to DFS (depth-first search) to optimize the generation path and accelerate the inference process.

[0075] Figure 2 It is a flowchart of a tree-structured speculative sampling method for generative model inference shown according to another exemplary embodiment. Figure 2 The shown process 20 is a detailed description of the application of the technology of the present application in the Medusa model.

[0076] As Figure 2 shown, in S202, the input sequence.

[0077] In S204, the input sequence is sorted in DFS. The optimization acceleration method based on DFS sorting of the present application can be added to the generative model inference. DFS sorting is used to adjust the positions of draft tokens and increase the maximum continuous length. Generally, the input sequence is sorted in the BFS manner. In the present application, the arrangement of draft tokens in BFS can be changed to the arrangement in DFS.

[0078] In a specific embodiment, the original BFS arrangement: [You, ate, him, me, apple, pear, of, of, ate, apple, pear, apple, pear, ate] "Him", "me", "apple", "pear" are all in the first layer. Since the previous token of "me" is "ate", which separates "him", and there is already a mask in this line, so "me" is not continuous, only "him" is continuous.

[0079] Changed to the DFS arrangement: [You, ate, him, of, apple, ate, pear, me, of, apple, pear, apple, ate, pear] In S206, the generative model performs inference, calculates KV, and caches them.

[0080] Figure 3 It is a schematic diagram of a tree-structured speculative sampling method for generative model inference shown according to another exemplary embodiment. As Figure 3 shown, in this arrangement, the largest continuous sequence is "You ate him of apple ate", and the maximum number of continuous draft tokens is increased from 1 to 4.

[0081] Although the last accepted sequence is "You ate his pear", the last number of draft tokens accepted is k = 3, and the maximum number of consecutive accepted tokens is p = 2. This way, one more token is cached, and one less token is calculated next time.

[0082] In actual testing, it is found that the draft token with the highest probability tends to be accepted. In other words, the leftmost column is likely to be directly accepted. The DFS traversal method of this application puts the chain that is most likely to be accepted in front.

[0083] In S208, sampling is performed to accept k draft tokens, of which p tokens are continuous with the input sequence.

[0084] In S210, the medusa model performs speculation and generates n draft tokens.

[0085] In S212, the uncached draft token and p tokens are cached.

[0086] In S214, the remaining kp tokens, a new token, and the n draft tokens output by the medusa model are added to the input sequence to generate the current sequence, and reasoning is continued based on the current sequence.

[0087] Figure 4 The present invention is a flowchart of a tree-structured speculative sampling method for generative model reasoning according to another exemplary embodiment. Figure 4 The process 40 shown is a detailed description of the application of the technology of the present application in the draft model. The draft model is smaller in scale than the target model and is mainly used to assist in generating preliminary candidate tokens.

[0088] like Figure 4 As shown, in S402, a sequence is input.

[0089] In S404, the draft model is inferred, and KV is calculated and stored.

[0090] In S406, the sampling is repeated multiple times until n draft tokens are generated.

[0091] Since the draft model is smaller in size than the target model, it is generated faster and has a relatively lighter computational burden. Therefore, when switching from BFS to DFS, running the draft model for a few more tokens has less impact on the overall performance.

[0092] In S408, DFS sorting is performed.

[0093] In a specific embodiment, the original BFS arrangement is: [You, ate, him, my, apple, pear, of, of, apple, pear, apple, pear, of] Change to DFS arrangement: [You ate his apple, pear, my apple, pear, apple, pear] In this switch, the order of "you ate him" does not change. So the key of "he" in the draft model can be retained.

[0094] In order to cooperate with the reasoning process of the target model, the draft model needs to make the following "concessions": Restriction on decoding order: During the generation process, model reasoning is decoded layer by layer, that is, each time a token must be generated before the next token can be generated. This makes the draft tokens decoded by the draft model necessarily arranged in BFS order. Correspondingly, its KVCache information is also stored in BFS order.

[0095] Impact of the conversion from BFS to DFS: When switching from BFS order to DFS order, the KV Cache information originally stored in the draft model according to BFS will become invalid. To support the DFS reasoning order of the target model, the draft model needs to regenerate some tokens and update the KV Cache.

[0096] In S410, the generative model performs reasoning, calculates KV and caches it.

[0097] In S412, sampling is performed and k draft tokens are received, of which p tokens are continuous with the input sequence.

[0098] In S414, the uncached draft token and p tokens are cached.

[0099] In S416, the remaining kp tokens and a new token are added to the input sequence to generate the current sequence, and reasoning is continued based on the current sequence.

[0100] In a specific application, the specific details of the storage parameters involved in this application are as follows: For the tree interface of draft tokens, combined with the order of sequence storage and the need for mask positions, we store the following parameters: draft parents: The relative position of the parent node in the draft tokens. The relative position of the root node is fixed to -1. draft levels: The level where the current node is located.

[0101] Parameters under BFS arrangement: Parameters under DFS arrangement: The generation of the mask can be easily implemented using draft parents. First, convert the relative position to the absolute position in the sequence. The mask position of each draft token is equal to the sum of all mask positions of the parent node and all the gaps from this node to the parent node. (This is a dynamic programming technique) For example: The arrangement in BFS mode is as follows: For "apple" at relative position 7, its parent node is "of" at relative position 4.

[0102] Mask position of "of": [1, 2, 3] Gaps from 4 to 7: [5, 6] Mask position of "apple": [1, 2, 3, 5, 6] Another example: The arrangement in DFS mode is as follows: For "pear" at relative position 8, its parent node is "of" at relative position 6.

[0103] Mask position of "of": [0, 1, 2, 3, 4] Gaps from 6 to 8: [7] Mask position of "pear": [0, 1, 2, 3, 4, 7] Using the design of draft parents, as long as starting from the first draft token, if the parent node of each subsequent token is equal to the previous token, the count is incremented by 1 until the first non - conforming condition is encountered and the counting stops. In this way, the maximum length of the identical sequence can be obtained in advance. There is no need to save the BFS and DFS token sequences.

[0104] For example: BFS draft parents: [-1, -1, -1, -1, 0, 1, 2] Maximum continuous sequence length: 1 BFS draft parents: [-1, 0, 0, 0, 1, 2] Maximum continuous sequence length: 2 BFS draft parents: [-1, 0, 1, 2, 2, 3, 4] Maximum continuous sequence length: 4 Under the condition of BFS, if the parent node of a token a is equal to the previous node b, it means that the depth of a is 1 higher than the depth of b. At the same time, because BFS has traversed to a level higher than the depth of b, it means that the level where b is located has been traversed. In other words, there is only one node b at the level where b is located.

[0105] In this application, we find out the parts of the continuous levels with only one node. This part of the tree can be degenerated into a chain. BFS is the same as DFS.

[0106] When encountering the first node whose parent node is not the previous node, because this will cause the generation of a mask (regardless of whether the parent node has a mask position, the gap from the node to the parent node must not be empty, so the mask position of the node exists), the node will definitely not be accepted by the kv cache (whether it is a draft model or a target model), because this will cause the generation of a mask (regardless of whether the parent node has a mask position, the gap from the node to the parent node must not be empty, so the mask position of the node exists).

[0107] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided in the present application are performed. The program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0108] In addition, it should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0109] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0110] Figure 5 is a block diagram of a tree-like speculative sampling device for generative model reasoning according to an exemplary embodiment. Figure 5 As shown, the tree-like speculative sampling device 50 for generative model reasoning includes: an input module 502, a reasoning module 504, an acceptance module 506, an update module 508, a recalculation module 510, and an output module 512.

[0111] The input module 502 is used to input the input sequence into the generative model; the input module 502 is also used to arrange the input sequence in the order of DFS traversal and input it into the generative model.

[0112] The inference module 504 is used for the generative model to perform inference calculation based on the input sequence to generate multiple candidate tokens in a tree structure; the inference module 504 is also used to cache kv cache information corresponding to the multiple candidate tokens.

[0113] The accepting module 506 is used to accept some candidate tokens in the tree structure; the accepting module 506 is also used to accept k candidate tokens in the tree structure, and cache the kv cache information of p tokens; and discard the kv cache information of kp tokens that are not cached in the part of the candidate tokens.

[0114] The update module 508 is used to update the input sequence through the partial candidate tokens to generate the current sequence; the update module 508 is also used to input the partial candidate tokens into the medusa speculation model to generate n candidate tokens; add kp tokens that are not cached and a new token to the input sequence; add n candidate tokens to the input sequence to generate the current sequence.

[0115] The updating module 508 is further configured to add kp tokens that are not cached and a new token to the input sequence to generate the current sequence.

[0116] The recalculation module 510 is used to perform the next reasoning calculation based on the current sequence and recalculate the uncached tokens in the next reasoning calculation; the recalculation module 510 is also used to recalculate the uncached kp tokens for the next reasoning calculation.

[0117] The output module 512 is used to output the generated final sequence when the termination condition is met.

[0118] According to the tree-like speculative sampling device for generative model reasoning of the present application, an input sequence is input into a generative model; the generative model performs reasoning calculation based on the input sequence to generate multiple candidate tokens in a tree structure; some candidate tokens in the tree structure are accepted; the input sequence is updated by the some candidate tokens to generate a current sequence; the next reasoning calculation is performed based on the current sequence, and the uncached tokens are recalculated in the next reasoning calculation; when the termination condition is met, the generated final sequence is output in a manner that can explore multiple possible generation paths in parallel, avoid the efficiency bottleneck caused by step-by-step reasoning, save computing resources in the process of reasoning calculation, and achieve a balance between cache efficiency and flexibility.

[0119] Figure 6It is a block diagram of an electronic device according to an exemplary embodiment.

[0120] Refer to the following Figure 6 The electronic device 600 according to this embodiment of the present application is described. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0121] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0122] The storage unit stores a program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 610 can perform the following steps: Figure 1 , Figure 2 , Figure 4 Follow the steps shown in .

[0123] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0124] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0125] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0126] The electronic device 600 may also communicate with one or more external devices 600' (e.g., keyboards, pointing devices, Bluetooth devices, etc.) so that a user can communicate with the device that interacts with the electronic device 600, and / or the electronic device 600 can communicate with any device that communicates with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. In addition, the electronic device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0127] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 7 As shown, the technical solution according to the implementation mode of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation mode of the present application.

[0128] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0129] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0130] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0131] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by a device, the computer-readable medium implements the following functions: inputting an input sequence into a generative model; the generative model performs reasoning calculation based on the input sequence to generate multiple candidate tokens in a tree structure; accepting some candidate tokens in the tree structure; updating the input sequence through the some candidate tokens to generate a current sequence; performing the next reasoning calculation based on the current sequence, and recalculating the tokens that are not cached in the next reasoning calculation; when the termination condition is met, outputting the generated final sequence.

[0132] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0133] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present application.

[0134] The exemplary embodiments of the present application are specifically shown and described above. It should be understood that the present application is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present application is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A tree-like speculative sampling method for generative model reasoning, characterized in that: include: Feed the input sequence into the generative model; The generative model performs reasoning calculation based on the input sequence to generate multiple candidate tokens in a tree structure; Accepting some candidate tokens in the tree structure; Updating the input sequence by using the partial candidate tokens to generate a current sequence; Perform the next inference calculation based on the current sequence, and recalculate the uncached token in the next inference calculation; When the termination condition is met, the final sequence generated is output.

2. The method according to claim 1, characterized in that Feed the input sequence into the generative model, including: The input sequence is arranged in the order of DFS traversal and input into the generative model.

3. The method according to claim 2, characterized in that Arranging the input sequence in the order of DFS traversal and inputting it into the generative model includes: Inputting the input sequence into the draft model to generate multiple candidate tokens; Cache the kv cache information corresponding to the multiple candidate tokens; The multiple candidate tokens are arranged in the order of DFS traversal and input into the generative model.

4. The method according to claim 3, characterized in that The original sequence is input into the draft model to generate multiple candidate tokens, including: The original sequence is input into the draft model multiple times, and the multiple candidate tokens are generated through multiple speculative sampling processes.

5. The method according to claim 1, characterized in that The generative model performs reasoning calculation based on the input sequence to generate multiple candidate tokens in a tree structure, and further includes: The kv cache information corresponding to the multiple candidate tokens is cached.

6. The method according to claim 1, characterized in that Accept some candidate tokens in the tree structure, including: Accept k candidate tokens in the tree structure and cache the kv cache information of p tokens; The kv cache information of the kp tokens that are not cached in the part of candidate tokens is discarded.

7. The method according to claim 6, characterized in that Accepting k candidate tokens in the tree structure and caching the kv cache information of p tokens, including: Accept k candidate tokens in the tree structure; Cache the kv cache of the first p consecutive tokens among the k candidate tokens.

8. The method according to claim 7, characterized in that Updating the input sequence by using the partial candidate tokens to generate a current sequence includes: Input the part of candidate tokens into the medusa speculation model to generate n candidate tokens; Add kp uncached tokens and a new token to the input sequence; Add n candidate tokens to the input sequence to generate the current sequence.

9. The method according to claim 7, characterized in that Updating the input sequence by using the partial candidate tokens to generate a current sequence includes: Add kp uncached tokens and a new token to the input sequence to generate the current sequence.

10. The method according to claim 1, characterized in that In the next step of inference calculation, the uncached token is recalculated, including: Calculate the uncached kp tokens again for the next step of inference calculation.

11. A tree-like speculative sampling device for generative model reasoning, characterized in that: include: An input module, which is used to input the input sequence into the generative model; An inference module, used for the generative model to perform inference calculation based on the input sequence to generate multiple candidate tokens in a tree structure; An accepting module, used for accepting some candidate tokens in the tree structure; An updating module, configured to update the input sequence using the partial candidate tokens to generate a current sequence; The recalculation module is used to perform the next inference calculation based on the current sequence and recalculate the uncached tokens in the next inference calculation; The output module is used to output the generated final sequence when the termination condition is met.

Images