Large model reasoning method and system based on cache-assisted parallel speculative decoding

Abstract

The invention relates to the technical field of large language models, in particular to a large model reasoning method and system based on cache-assisted parallel speculation decoding. The method comprises the following steps: placing a first candidate word generated by a first draft model into a cache region; obtaining a second candidate word generated by a second draft model; querying a first target candidate word hit by the second candidate word in the candidate word sequence; taking the second candidate word and the first target candidate word as a lexical element sequence of the next second reasoning period of the second draft model; placing a second candidate word obtained by a second draft model and the first candidate word into the cache region; reasoning lexical elements are obtained; querying a second target candidate word in the candidate word sequence; and taking the inference lemma and the second target candidate word as a lemma sequence of the next third inference period of the target inference model. According to the target inference model, multiple inference lems can be obtained through one-time lem inference, and the inference speed of the target inference model is greatly increased.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular, to a large model reasoning method and system based on cache-assisted parallel speculative decoding. Background Art

[0002] With the groundbreaking development of the Transformer architecture, large language models (LLMs) such as GPT-4 and PaLM have been widely used in scenarios such as text generation and code completion. However, their autoregressive generation mechanism leads to significant inference latency—the model must generate tokens, or lemmas, one by one in sequence (N tokens require N forward computations, with each forward computation representing a lemma inference). This sharply conflicts with the demand for real-time interaction in practical applications. Due to the limitations of the autoregressive generation mechanism, several technologies exist to accelerate the generation process.

[0003] The acceleration methods in the prior art and their shortcomings are as follows:

[0004] First, model compression: quantization, pruning, and other methods are used to reduce computational complexity, but this can lead to a loss of model quality.

[0005] Second, hardware acceleration: using dedicated chips (such as TPU v4) to improve throughput, but the hardware cost has increased sharply, which is costly. Summary of the Invention

[0006] The present application provides a large model reasoning method and system based on cache-assisted parallel speculative decoding to solve one or more technical problems existing in the prior art and at least provide a beneficial choice or create conditions.

[0007] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0008] According to one aspect of an embodiment of the present application, a large model inference method based on cache-assisted parallel speculative decoding is provided, which is applied to a large language model. The large language model includes a cache area and a first drafting model, a second drafting model, and a target inference model for performing word-meta inference. The word-meta inference speeds are, from fastest to slowest, the first drafting model, the second drafting model, and the target inference model. The method includes:

[0009] placing a first candidate word generated by the first drafting model after a first reasoning cycle into the cache to update a candidate word sequence in the cache;

[0010] obtaining a second candidate word generated by the second drafting model after the second reasoning cycle;

[0011] If the second candidate word hits the candidate word sequence in the cache area, searching the candidate word sequence for the first target candidate word hit by the second candidate word;

[0012] using the second candidate word and the first target candidate word as a word-gram sequence for a second reasoning cycle of the second drafting model;

[0013] During the next second reasoning cycle, placing the second candidate word obtained by the second drafting model according to the word-gram sequence and the first candidate word generated by the first drafting model into the cache area to update the candidate word sequence;

[0014] Obtaining an inference token generated by the target inference model after the third inference cycle;

[0015] If the inference word unit hits the candidate word sequence, searching the candidate word sequence for a second target candidate word hit by the inference unit;

[0016] Using the inference word-gram and the second target candidate word as a word-gram sequence for a third inference cycle of the target inference model;

[0017] The duration of the first reasoning cycle is shorter than the duration of the second reasoning cycle, and the duration of the second reasoning cycle is shorter than the duration of the third reasoning cycle.

[0018] In one embodiment of the present application, based on the aforementioned scheme, the first drafting model, the second drafting model and the target reasoning model perform word-meta reasoning steps in parallel. During the process of performing word-meta reasoning by the first drafting model, the first drafting model generates a first candidate word after the first reasoning cycle, and the first drafting model generates multiple first candidate words after the second reasoning cycle or the third reasoning cycle; wherein, the number of first candidate words of the first drafting model after the second reasoning cycle is less than the number of first candidate words of the first drafting model after the third reasoning cycle.

[0019] In one embodiment of the present application, based on the aforementioned scheme, when the second drafting model performs word-unit reasoning, the second drafting model generates a second candidate word after the second reasoning cycle, and the second drafting model generates multiple second candidate words after the third reasoning cycle.

[0020] In one embodiment of the present application, based on the aforementioned scheme, when the target reasoning model performs word-unit reasoning, the target reasoning model generates an inference word-unit after the third reasoning cycle, and the first drafting model generates multiple first candidate words after the third reasoning cycle and places the generated multiple first candidate words into the cache area for word-unit sequence judgment; the second drafting model generates multiple second candidate words after the third reasoning cycle and places the generated multiple second candidate words into the cache area for word-unit sequence judgment.

[0021] In one embodiment of the present application, based on the above solution, after searching the candidate word sequence for the first target candidate word hit by the second candidate word, the method further includes:

[0022] Query whether there is a candidate word that is not matched by the second candidate word;

[0023] If there is a candidate word that does not match the second candidate word, the candidate word that does not match the second candidate word is deleted from the cache area.

[0024] In one embodiment of the present application, based on the above solution, the method further includes:

[0025] During the word-gram reasoning process of the second drafting model in the next second reasoning cycle according to the word-gram sequence of the second reasoning cycle, determining whether the first target candidate word in the word-gram sequence hits the candidate word sequence;

[0026] If the first target candidate word in the word-gram sequence hits the candidate word sequence, searching the candidate word sequence for a third target candidate word hit by the first target candidate word and the second candidate word;

[0027] The first target candidate word, the second candidate word and the third target candidate word are combined into a word-gram sequence of the next second reasoning cycle.

[0028] In one embodiment of the present application, based on the above solution, the method further includes:

[0029] During the word-gram inference process of the target inference model in the next third inference cycle according to the word-gram sequence of the third inference cycle, if the second target candidate word in the word-gram sequence hits the candidate word sequence, searching the candidate word sequence for a fourth target candidate word hit by the second target candidate word and the inference word-gram;

[0030] Combining the second target candidate word, the inference word-gram and the fourth target candidate word into a word-gram sequence of the next third inference cycle;

[0031] The word-gram sequence of the third reasoning cycle is obtained according to the word-gram sequence of the second reasoning cycle.

[0032] According to one aspect of an embodiment of the present application, a large model inference system based on cache-assisted parallel speculative decoding is proposed, which is applied to a large language model. The large language model includes a cache area and a first drafting model, a second drafting model, and a target inference model for performing word-meta inference. The word-meta inference speeds are, from fastest to slowest, the first drafting model, the second drafting model, and the target inference model. The system includes:

[0033] a first inference unit, configured to place a first candidate word generated by the first drafting model after a first inference cycle into the cache area, so as to update a candidate word sequence in the cache area;

[0034] a first acquiring unit, configured to acquire a second candidate word generated by the second drafting model after the second reasoning cycle;

[0035] a first query unit, configured to query the candidate word sequence for a first target candidate word hit by the second candidate word if the second candidate word hits the candidate word sequence in the cache area;

[0036] a second reasoning unit, configured to use the second candidate word and the first target candidate word as a word-gram sequence for a second reasoning cycle of the second drafting model;

[0037] an updating unit, configured to place the second candidate word obtained by the second drafting model according to the word-gram sequence and the first candidate word generated by the first drafting model into the cache area during a next second reasoning cycle, so as to update the candidate word sequence;

[0038] A second acquiring unit, configured to acquire an inference word generated by the target inference model after a third inference cycle;

[0039] a second query unit, configured to query the candidate word sequence for a second target candidate word hit by the inference dimensional unit if the inference dimensional unit hits the candidate word sequence;

[0040] a third inference unit, configured to use the inference word-gram and the second target candidate word as a word-gram sequence for a next third inference cycle of the target inference model;

[0041] The duration of the first reasoning cycle is shorter than the duration of the second reasoning cycle, and the duration of the second reasoning cycle is shorter than the duration of the third reasoning cycle.

[0042] The implementation principles of this application are as follows:

[0043] First, the first drafting model, the second drafting model and the target reasoning model perform word-meta reasoning in parallel, but the first drafting model, the second drafting model and the target reasoning model have different word-meta reasoning speeds. The first drafting model has the fastest reasoning speed. Therefore, in the same amount of time, the number of first candidate words inferred by the first drafting model is greater than the number of second candidate words inferred by the second drafting model and the number of inferred words by the target reasoning model.

[0044] The reasoning speed of the second drafting model is faster than that of the target reasoning model. Therefore, in the same amount of time, the number of second candidate words inferred by the second drafting model is greater than the number of inferred word units of the target reasoning model.

[0045] In word-unit reasoning, the faster the reasoning speed, the worse the quality of the candidate words obtained by reasoning. Therefore, this application speeds up the word-unit reasoning speed of the target reasoning model through a first drafting model and a second drafting model with different reasoning capabilities.

[0046] Specifically, the first drafting model is only responsible for generating the first candidate word and placing the first candidate word into the cache, so that the candidate word sequence in the cache is updated. The first drafting model serves the second drafting model. During the word-meta reasoning process, the second drafting model determines whether the second candidate word it generates hits the first candidate word and / or second candidate word generated by the first drafting model and / or the second drafting model in the cache. If so, the second drafting model will form the word-meta sequence of the next word-meta reasoning process with the second candidate word generated and the hit first target candidate word, and continue reasoning.

[0047] Furthermore, the second candidate word inferred by the second drafting model and the formed word-gram sequence serve the target reasoning model. If the inference word-gram generated by the target reasoning model hits the candidate word sequence, since the candidate word sequence is jointly maintained and updated by the first drafting model and the second drafting model, then if the inference word-gram hits the candidate word sequence and obtains the second target candidate word, a new word-gram sequence is formed at this time, that is, the word-gram sequence corresponding to the target reasoning model includes the inference word-gram generated by the target reasoning model itself and the second target candidate word successfully verified by the second drafting model after querying the cache area. In this way, compared with the existing target reasoning model that can only generate one inference word-gram at a time, the word-gram sequence of the present application can include the inference word-gram and the second target candidate word, that is, multiple inference word-grams for reasoning in the next third reasoning cycle, which speeds up the reasoning speed of the target reasoning model.

[0048] The beneficial effects of the present application are as follows: the first drafting model quickly and continuously generates the first candidate word, the second drafting model verifies the first candidate word generated by the first drafting model, and after successful verification, carries out word-gram reasoning of the next second reasoning cycle with the successfully verified first target candidate word. In the process of continuously performing word-gram reasoning, after the target reasoning model completes a word-gram reasoning, that is, after generating reasoning words after the third reasoning cycle, the word-gram sequence carried by the second drafting model, that is, the word-gram sequence obtained by the historical word-gram reasoning process, can be combined with the reasoning words to form the word-gram sequence of the target reasoning model in the next third reasoning cycle, so that the target reasoning model can obtain multiple reasoning words (the second target candidate word can also be used as a reasoning word) by performing a word-gram reasoning, which greatly improves the reasoning speed of the target reasoning model.

[0049] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, explaining the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings: Figure 1 Flowchart of a large model inference method based on cache-assisted parallel speculative decoding according to an embodiment of the present application; Figure 2 A logical diagram of parallel reasoning shown in an embodiment of the present application; Figure 3 This is an architectural diagram of a large language model shown in an embodiment of the present application; Figure 4 A block diagram of a large model inference system based on cache-assisted parallel speculative decoding according to an embodiment of the present application; Figure 5 1 is a structural diagram of an electronic device according to the present application.

[0051] Figure 1 Flowchart of a large model inference method based on cache-assisted parallel speculative decoding according to an embodiment of the present application;

[0052] Figure 2 A logical diagram of parallel reasoning shown in an embodiment of the present application;

[0053] Figure 3 This is an architectural diagram of a large language model shown in an embodiment of the present application;

[0054] Figure 4 This is a block diagram of a large model inference system based on cache-assisted parallel speculative decoding according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0056] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0057] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or micro-controller node devices.

[0058] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0059] It should be noted that the term "plurality" used in this document refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0060] The following is a detailed explanation of the background technology of the embodiments of the present application:

[0061] With the groundbreaking development of the Transformer architecture, large language models (LLMs) such as GPT-4 and PaLM have been widely used in scenarios such as text generation and code completion. However, their autoregressive generation mechanism leads to significant inference latency—the model must generate tokens, or lemmas, one by one in sequence (N tokens require N forward computations, with each forward computation representing a lemma inference). This sharply conflicts with the demand for real-time interaction in practical applications. Due to the limitations of the autoregressive generation mechanism, several technologies exist to accelerate the generation process.

[0062] The acceleration methods in the prior art and their shortcomings are as follows:

[0063] First, model compression: quantization, pruning, and other methods are used to reduce computational complexity, but this can lead to a loss of model quality.

[0064] Second, hardware acceleration: Using specialized chips (such as the TPU v4) to increase throughput, but this can lead to a surge in hardware costs and high costs.

[0065] The larger the drafting model, the better the drafting effect, the easier it is to be accepted by the target model, the longer the drafting model reasoning time, and the longer the corresponding target model waiting time. Although a longer word sequence is accepted each time, the time the target model waits for the drafting model to draft becomes longer. The smaller the drafting model, the worse the drafting effect, the less likely it is to be accepted by the target model, the shorter the drafting model reasoning time, and the corresponding target model waiting time is shorter. Although the waiting time for drafting becomes shorter, the length of the word sequence accepted by the target model each time becomes shorter. This application improves the accelerated reasoning effect of large language models by balancing the size of the drafting model.

[0066] The following is a detailed description of the implementation details of the technical solution of the embodiment of the present application:

[0067] According to one aspect of the present application, a large model reasoning method based on cache-assisted parallel speculative decoding is provided. Figure 1 Flowchart of a large model inference method based on cache-assisted parallel speculative decoding according to an embodiment of the present application. The large model inference method based on cache-assisted parallel speculative decoding includes at least steps S1 to S8, which are described in detail as follows:

[0068] In step S1, a first candidate word generated by the first drafting model after a first reasoning cycle is placed into the cache to update the candidate word sequence in the cache;

[0069] In step S2, obtaining a second candidate word generated by the second drafting model after the second reasoning cycle;

[0070] In step S3, if the second candidate word hits the candidate word sequence in the cache area, the first target candidate word hit by the second candidate word is searched in the candidate word sequence;

[0071] In step S4, the second candidate word and the first target candidate word are used as a word-gram sequence for a second reasoning cycle of the second drafting model;

[0072] In step S5, during the next second reasoning cycle, the second candidate word obtained by the second drafting model according to the word-gram sequence and the first candidate word generated by the first drafting model are placed in the cache to update the candidate word sequence;

[0073] In step S6, the inference word-gram generated by the target inference model after the third inference cycle is obtained;

[0074] In step S7, if the inference word unit hits the candidate word sequence, the candidate word sequence is searched for the second target candidate word hit by the inference unit;

[0075] In step S8, the inference word-gram and the second target candidate word are used as a word-gram sequence for the next third inference cycle of the target inference model;

[0076] The duration of the first reasoning cycle is shorter than the duration of the second reasoning cycle, and the duration of the second reasoning cycle is shorter than the duration of the third reasoning cycle.

[0077] The implementation principles of this application are as follows:

[0078] First, the first drafting model, the second drafting model and the target reasoning model perform word-meta reasoning in parallel, but the first drafting model, the second drafting model and the target reasoning model have different word-meta reasoning speeds. The first drafting model has the fastest reasoning speed. Therefore, in the same amount of time, the number of first candidate words inferred by the first drafting model is greater than the number of second candidate words inferred by the second drafting model and the number of inferred words by the target reasoning model.

[0079] The reasoning speed of the second drafting model is faster than that of the target reasoning model. Therefore, in the same amount of time, the number of second candidate words inferred by the second drafting model is greater than the number of inferred word units of the target reasoning model.

[0080] In word-meta reasoning, the faster the reasoning speed, the lower the quality of the candidate words obtained by reasoning. Therefore, this application uses a first drafting model and a second drafting model with different reasoning capabilities to accelerate the word-meta reasoning speed of the target reasoning model. It should be noted that the execution subject of the large-model reasoning method based on cache-assisted parallel speculative decoding provided by this application is the server.

[0081] Specifically, the first drafting model is only responsible for generating the first candidate word and placing the first candidate word into the cache, so that the candidate word sequence in the cache is updated. The first drafting model serves the second drafting model. During the word-meta reasoning process, the second drafting model determines whether the second candidate word it generates hits the first candidate word and / or second candidate word generated by the first drafting model and / or the second drafting model in the cache. If so, the second drafting model will form the word-meta sequence of the next word-meta reasoning process with the second candidate word generated and the hit first target candidate word, and continue reasoning.

[0082] Furthermore, the second candidate word inferred by the second drafting model and the formed word-gram sequence serve the target reasoning model. If the inference word-gram generated by the target reasoning model hits the candidate word sequence, since the candidate word sequence is jointly maintained and updated by the first drafting model and the second drafting model, then if the inference word-gram hits the candidate word sequence and obtains the second target candidate word, a new word-gram sequence is formed at this time, that is, the word-gram sequence corresponding to the target reasoning model includes the inference word-gram generated by the target reasoning model itself and the second target candidate word successfully verified by the second drafting model after querying the cache area. In this way, compared with the existing target reasoning model that can only generate one inference word-gram at a time, the word-gram sequence of the present application can include the inference word-gram and the second target candidate word, that is, multiple inference word-grams, so as to perform reasoning in the next third reasoning cycle, thereby speeding up the reasoning speed of the target reasoning model.

[0083] The first drafting model quickly and continuously generates first candidate words. The second drafting model verifies the first candidate words generated by the first drafting model. After successful verification, the second drafting model carries the successfully verified first target candidate words for word-meta inference in the next second reasoning cycle. During the continuous word-meta inference process, after the target reasoning model completes a word-meta inference, that is, after generating inference words after the third reasoning cycle, the word-meta sequence carried by the second drafting model (that is, the word-meta sequence obtained from the historical word-meta inference process) can be combined with the inference words to form the word-meta sequence of the target reasoning model in the next third reasoning cycle. This allows the target reasoning model to obtain multiple inference words after a single word-meta inference (the second target candidate word can also be used as an inference word), greatly improving the reasoning speed of the target reasoning model.

[0084] This application adopts a new architecture based on cache and branch prediction (i.e., three different branches of the first drafting model, the second drafting model, and the target reasoning model). By reasoning the first drafting model, the second drafting model, and the target reasoning model in parallel, the "draft first, then verify" paradigm is improved to "continuous verification". The problem of the target reasoning model waiting in speculative decoding is solved. Through the caching strategy, multiple drafting models with different model capabilities are allowed to collaborate (i.e., the first drafting model and the second drafting model) to draft (i.e., perform word element reasoning), which solves the problem of poor quality of candidate words (candidates) generated by a single drafting model. Through these two strategies, the target model can simultaneously increase the number of reasoning tokens accepted for each reasoning and the frequency of reasoning of the target reasoning model. There is no need to wait for the reasoning time of the first drafting model and the second drafting model, and continuous reasoning can be performed, which greatly speeds up the reasoning speed of reasoning decoding, while avoiding the problem of GPU idling during the waiting process and avoiding waste of computing resources.

[0085] Specific as Figure 2 As shown, the entire process of the target reasoning model, the first drafting model and the second drafting model assisting each other in reasoning is described, taking the correct reasoning sequence of ABCDEFG... as an example. Assume that the time for a single forward conduction process of the first drafting model is t, the forward conduction process is the word element reasoning process described in this application, and the time for a single forward conduction process of the second drafting model is 3t. The time for a single forward conduction process of the target reasoning model is 9t. That is to say, the first drafting model can generate 9 candidate words (first candidate words) within 9t, the second drafting model can generate 3 candidate words (second candidate words) within 9t, and the target reasoning model can generate one reasoning word element within 9t.

[0086] Among them, the first reasoning cycle, the second reasoning cycle and the third reasoning cycle corresponding to the present application are t, 3t and 9t respectively in this example. It should be noted that the first reasoning cycle is the reasoning cycle of the first drafting model, the second reasoning cycle is the reasoning cycle of the second drafting model, and the third reasoning cycle is the reasoning cycle of the target reasoning model. Then the duration of the first reasoning cycle is the shortest, which is t (the value of t can be set arbitrarily); the duration of the second reasoning cycle is greater than the duration of the third reasoning cycle and less than the duration of the first reasoning cycle, which is 3t (the value of t can be set arbitrarily); the duration of the third reasoning cycle is the longest, which is 9t (the value of t can be set arbitrarily). It should be noted that the first reasoning cycle, the second reasoning cycle and the third reasoning cycle have their own time lines and do not belong to the same time line. That is to say, the first drafting model, the second drafting model and the target reasoning model execute the word-unit reasoning steps in parallel. From this, it can be seen that the first drafting model will generate the first candidate word in multiple first reasoning cycles such as t, 2t, 3t....nt (n is any positive integer greater than 3), the second drafting model will generate the second candidate word in multiple second reasoning cycles such as 3t, 6t, 9t....3nt (n is any positive integer greater than 3), and the target reasoning model will generate reasoning word units in multiple third reasoning cycles such as 9t, 18t, 27t....9nt (n is any positive integer greater than 3). For example, at the moment of 9t, the first drafting model, the second drafting model and the target reasoning model will all generate candidate words / inference word units, i.e. tokens.

[0087] The primary function of the first drafting model is to store the inference results (i.e., the first candidate word) in a cache, providing candidate words for the second drafting model and the target inference model. By 3 t, the first drafting model had inferred three tokens (i.e., the first candidate word, which can also be used as a token): A, B, and f, and stored them all in the cache.

[0088] At this time, the second drafting model has just completed a forward conduction process. Then the second drafting model carries the inference result A to query the cache. The cache hits B, f. Reorganize its own word sequence into ABf, and carry ABf to perform the next reasoning (that is, the word reasoning of the next second reasoning cycle described in this application). By 6t, the first drafting model continues to infer the sequence egh and put it into the cache. The second forward conduction process of the second drafting model ends, and the inference results are speculated, sampled and verified. Token B is accepted and token f is rejected, and the next token C is sampled according to the logits of token B. Token C is placed in the cache, and the first drafting model is notified to update the sequence, that is, fegh in the first drafting model is deleted. Among them, token C is the third target candidate word.

[0089] The second drafting model continues to reason with ABC in the next second reasoning cycle. The first drafting model receives the update notification from the second drafting model and first rolls back the kv-cache of the current sequence to the position of sequence AB. The first drafting model then carries ABC in the next reasoning (i.e., the next first reasoning cycle).

[0090] As of time 9t. The first drafting model generates a new candidate sequence DEF due to carrying ABC and puts it into the cache. The second drafting model generates a candidate word D and puts it into the cache. The target reasoning model now generates the first verified token A. The target reasoning model queries the cache and hits the candidate word sequence in the cache (i.e., A hits the aforementioned ABCDEF). Then, the target reasoning model carries the word sequence ABCDEF for reasoning in the next reasoning (the next third reasoning cycle).

[0091] The second drafting model queries the cache and carries EF for reasoning (because ABCD has been inferred before). As of 18t, the target reasoning model has completed the second forward process. The results are speculated and sampled. The sequence BCDEF is accepted, and the next token G is sampled according to the logits of F. At this time, one forward process generates 6 tokens, and the target reasoning model is accelerated by 6 times. Token G is the fourth target candidate word. The cache is updated after the reasoning is completed, and the erroneous cache sequence ABfegh (that is, tokens that have not been hit or verified successfully) is deleted from the cache. This cycle continues until the reasoning limit is reached or the end word is encountered. It should be noted that Figure 2 The cache in is the cache area.

[0092] Furthermore, two drafting models are used to accelerate the target inference model. The two drafting models (i.e., the first drafting model and the second drafting model) are loaded onto different GPU devices respectively to prevent resource competition when they are loaded onto the same GPU.

[0093] Load the target inference model to the target model server cluster. Among them, the first drafting model, the second drafting model and the target model communicate through nvlink or PCIE channels. After the drafting model is loaded, a cache area is created to store the reasoning verification results of the drafting model. Logically speaking, the small drafting model (first drafting model) exists to speed up the large draft model (second drafting model). This is to ensure the speed of generating candidate words for the large draft model. At the same time, the quality of the candidate words is guaranteed. From the perspective of the target inference model, it is to maintain a candidate sequence with a longer window length and higher candidate word quality.

[0094] The draft model obtains the request and converts the request into a tokenid. The tokenid is input into both draft models for the prefill phase. The filling phase processes all tokens of the request at once to form the initial verification success sequence. The initial verification success sequence is Figure 2 Because the two drafting models have different inference speeds, each drafting model performs its own inference process without synchronization. After prefill execution is completed, the subsequent inference process proceeds directly without waiting. That is, there is no waiting process between the first drafting model and the second drafting model. The cache table is maintained through asynchronous communication.

[0095] During the inference phase, we first determine whether there are any rejected tokens after the target inference model completes inference. If so, we need to roll back the kv-cache to the sequence length verified by the large language model. The kv-cache that exceeds this length will be cleared.

[0096] The second draft model performs reasoning: First, it queries the cache maintained jointly with the first model to check for a hit. If the sequence hits, it carries on reasoning with the candidate word; if not, it proceeds directly to reasoning. The first draft model performs reasoning: Direct reasoning.

[0097] Put the inferred tokens into the cache.

[0098] Check whether there is a common token prefix in the buffer, and add the generated tokens to the common prefix to generate a new sequence. For example, Figure 2 As shown, the first drafting model produces ABf, and the second drafting model produces A. Then it can be considered that the second drafting model A has a common prefix with ABf of the first drafting model. At this time, the second drafting model is also considered to have hit the first target candidate word ABf.

[0099] like Figure 3 As shown, Figure 3 For the overall architecture of the large language model of this application, the word sequence generated by the first drafting model and the second drafting model is provided to the target reasoning model after verification, so that the reasoning speed of the target reasoning model is accelerated.

[0100] This application organizes a cache area as a bridge between the two drafting models and the target reasoning model, cleverly decoupling the dependency between the two drafting models and the target reasoning model, thereby realizing the parallel operation of the drafting model and the target reasoning model, saving the drafting time of traditional speculative decoding waiting for the drafting model during the reasoning process, and improving the utilization of GPU resources.

[0101] Compared with traditional inference decoding, this application uses two drafting models with different capabilities and inference speeds, and uses a small drafting model (the first drafting model) to accelerate the inference speed of the large drafting model (the second drafting model). While improving the depth of the candidate word sequence, it also ensures the quality of the candidate words (the quality of the word unit sequence). At the same time, the two drafting models can ensure the width of the candidate words, so that the target model can perform inference decoding with a longer window and higher quality candidate words (inference word units).

[0102] like Figure 4 As shown, Figure 4 This is a system block diagram of a large model reasoning system 300 based on cache-assisted parallel speculative decoding provided in an embodiment of the present application. The large model reasoning system 300 based on cache-assisted parallel speculative decoding includes:

[0103] A first inference unit 301 is configured to place a first candidate word generated by the first drafting model after a first inference cycle into the cache to update a candidate word sequence in the cache;

[0104] A first acquiring unit 302 is configured to acquire a second candidate word generated by the second drafting model after the second reasoning cycle;

[0105] A first query unit 303 is configured to query the candidate word sequence for a first target candidate word hit by the second candidate word if the second candidate word hits the candidate word sequence in the cache area;

[0106] A second reasoning unit 304 is configured to use the second candidate word and the first target candidate word as a word-gram sequence for a second reasoning cycle of the second drafting model;

[0107] an updating unit 305 configured to place the second candidate word obtained by the second drafting model according to the word-gram sequence and the first candidate word generated by the first drafting model into the cache during a next second reasoning cycle, so as to update the candidate word sequence;

[0108] A second acquiring unit 306 is configured to acquire an inference word generated by the target inference model after a third inference cycle;

[0109] A second query unit 307 is configured to query the candidate word sequence for a second target candidate word hit by the inference dimensional unit if the inference dimensional unit hits the candidate word sequence;

[0110] A third inference unit 308, configured to use the inference word-gram and the second target candidate word as a word-gram sequence for a next third inference cycle of the target inference model;

[0111] The duration of the first reasoning cycle is shorter than the duration of the second reasoning cycle, and the duration of the second reasoning cycle is shorter than the duration of the third reasoning cycle.

[0112] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including, for example, several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0113] Furthermore, the above-mentioned figures are merely illustrative of the processes included in the methods according to exemplary embodiments of the present application and are not intended to be limiting. It is readily understood that the processes illustrated in the above-mentioned figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0114] It should be understood that the present application is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be performed without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A large model inference method based on cache-assisted parallel speculative decoding, characterized in that: Applied to a large language model, the large language model includes a cache area and a first drafting model, a second drafting model, and a target reasoning model for performing word-gram reasoning, wherein the speed of word-gram reasoning is, from fastest to slowest, the first drafting model, the second drafting model, and the target reasoning model; the method includes: placing a first candidate word generated by the first drafting model after a first reasoning cycle into the cache to update a candidate word sequence in the cache; obtaining a second candidate word generated by the second drafting model after the second reasoning cycle; If the second candidate word hits the candidate word sequence in the cache area, searching the candidate word sequence for the first target candidate word hit by the second candidate word; using the second candidate word and the first target candidate word as a word-gram sequence for a second reasoning cycle of the second drafting model; During the next second reasoning cycle, placing the second candidate word obtained by the second drafting model according to the word-gram sequence and the first candidate word generated by the first drafting model into the cache area to update the candidate word sequence; Obtaining an inference token generated by the target inference model after the third inference cycle; If the inference word unit hits the candidate word sequence, searching the candidate word sequence for a second target candidate word hit by the inference unit; Using the inference word-gram and the second target candidate word as a word-gram sequence for a third inference cycle of the target inference model; The duration of the first reasoning cycle is shorter than the duration of the second reasoning cycle, and the duration of the second reasoning cycle is shorter than the duration of the third reasoning cycle.

2. The large model inference method based on cache-assisted parallel speculative decoding according to claim 1, characterized in that The first drafting model, the second drafting model and the target reasoning model perform word-meta reasoning steps in parallel. During the process of performing word-meta reasoning by the first drafting model, the first drafting model generates a first candidate word after the first reasoning cycle, and the first drafting model generates multiple first candidate words after the second reasoning cycle or the third reasoning cycle; wherein, the number of first candidate words of the first drafting model after the second reasoning cycle is less than the number of first candidate words of the first drafting model after the third reasoning cycle.

3. The large model inference method based on cache-assisted parallel speculative decoding according to claim 1, characterized in that During the process of performing word-unit reasoning by the second drafting model, the second drafting model generates a second candidate word after the second reasoning cycle, and the second drafting model generates multiple second candidate words after the third reasoning cycle.

4. The large model inference method based on cache-assisted parallel speculative decoding according to claim 1, characterized in that During the process of performing word-gram reasoning by the target reasoning model, the target reasoning model generates an inference word-gram after the third reasoning cycle, and the first drafting model generates a plurality of first candidate words after the third reasoning cycle and places the generated plurality of first candidate words into a cache area for determining a word-gram sequence; The second drafting model generates a plurality of second candidate words after the third reasoning cycle and places the generated plurality of second candidate words into a cache area for determining a word unit sequence.

5. The large model inference method based on cache-assisted parallel speculative decoding according to claim 1, characterized in that After searching the candidate word sequence for a first target candidate word hit by the second candidate word, the method further includes: Query whether there is a candidate word that is not matched by the second candidate word; If there is a candidate word that does not match the second candidate word, the candidate word that does not match the second candidate word is deleted from the cache area.

6. The large model inference method based on cache-assisted parallel speculative decoding according to claim 1, characterized in that: The method further comprises: During the word-gram reasoning process of the second drafting model in the next second reasoning cycle according to the word-gram sequence of the second reasoning cycle, determining whether the first target candidate word in the word-gram sequence hits the candidate word sequence; If the first target candidate word in the word-gram sequence hits the candidate word sequence, searching the candidate word sequence for a third target candidate word hit by the first target candidate word and the second candidate word; The first target candidate word, the second candidate word and the third target candidate word are combined into a word-gram sequence of the next second reasoning cycle.

7. The large model inference method based on cache-assisted parallel speculative decoding according to claim 1, characterized in that The method further comprises: During the word-gram inference process of the target inference model in the next third inference cycle according to the word-gram sequence of the third inference cycle, if the second target candidate word in the word-gram sequence hits the candidate word sequence, searching the candidate word sequence for a fourth target candidate word hit by the second target candidate word and the inference word-gram; Combining the second target candidate word, the inference word-gram and the fourth target candidate word into a word-gram sequence of the next third inference cycle; The word-gram sequence of the third reasoning cycle is obtained according to the word-gram sequence of the second reasoning cycle.

8. A large model inference system based on cache-assisted parallel speculative decoding, characterized in that: Applied to a large language model, the large language model includes a cache area and a first drafting model, a second drafting model, and a target reasoning model for performing word-gram reasoning, wherein the speed of word-gram reasoning is, from fastest to slowest, the first drafting model, the second drafting model, and the target reasoning model; the system includes: a first inference unit, configured to place a first candidate word generated by the first drafting model after a first inference cycle into the cache area, so as to update a candidate word sequence in the cache area; a first acquiring unit, configured to acquire a second candidate word generated by the second drafting model after the second reasoning cycle; a first query unit, configured to query the candidate word sequence for a first target candidate word hit by the second candidate word if the second candidate word hits the candidate word sequence in the cache area; a second reasoning unit, configured to use the second candidate word and the first target candidate word as a word-gram sequence for a second reasoning cycle of the second drafting model; an updating unit, configured to place the second candidate word obtained by the second drafting model according to the word-gram sequence and the first candidate word generated by the first drafting model into the cache area during a next second reasoning cycle, so as to update the candidate word sequence; A second acquiring unit, configured to acquire an inference word generated by the target inference model after a third inference cycle; a second query unit, configured to query the candidate word sequence for a second target candidate word hit by the inference dimensional unit if the inference dimensional unit hits the candidate word sequence; a third inference unit, configured to use the inference word-gram and the second target candidate word as a word-gram sequence for a next third inference cycle of the target inference model; The duration of the first reasoning cycle is shorter than the duration of the second reasoning cycle, and the duration of the second reasoning cycle is shorter than the duration of the third reasoning cycle.

Images