Text generation method and device, storage medium equipment and program product

By combining speculative sampling and paging management methods in large-scale language models, the problems of slow speed and low throughput in the LLM inference process are solved, and more efficient text generation and system throughput are achieved.

CN119090015APending Publication Date: 2024-12-06ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411191242.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing large-scale language models (LLMs) have problems with slow speed and low throughput in the process of inference generation of natural language texts, especially due to the low computation and storage efficiency of attention information, which leads to the overall system throughput limited.

Method used

The method of speculative sampling is used to generate multiple alternative text sequences, and logical blocks are allocated in the key-value cache, and managed according to the criterion of logical block mapping as physical blocks. The attention information is used to accelerate the text generation process, and the video memory is optimized through paging management. Space utilization and improve system throughput.

Benefits of technology

Through the combination of speculative sampling and paging management, the LLM inference process is significantly accelerated, the number of cycles is reduced, the overall throughput of the system is improved, and the utilization of memory space and the utilization of hardware resources is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090015A_ABST
    Figure CN119090015A_ABST
Patent Text Reader

Abstract

The invention provides a text generation method and device, a storage medium device and a program product, which are applied to LLM, and in each cycle of LLM, a first text sequence after a current text sequence can be obtained through a speculation sampling method, and a plurality of alternative sequences are formed. And in the key value cache, distributing logic blocks for the text units in the plurality of alternative sequences, and mapping the logic blocks into physical blocks. In the physical block mapping process, in the alternative sequences obtained through speculative sampling, the logic blocks distributed for the same text unit are mapped into the same physical block. And determining a text unit newly generated in the loop through LLM by using attention information stored in the key value cache. Aiming at a special scene that a plurality of alternative sequences have the same text unit under the condition of combining speculation sampling and paging management, a first criterion that a logic block is mapped into a physical block is provided, so that the plurality of alternative sequences can occupy a video memory space as little as possible, and the utilization rate of the video memory space is increased.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer applications, and in particular, to a text generation method, apparatus, storage medium device, and program product. Background Art

[0002] Large Language Model (LLM) is a deep learning model trained on massive text data. It can understand the meaning of text and generate natural language text based on given text, and handle various natural language tasks such as text summarization, question answering, translation, etc.

[0003] However, the LLM in the prior art has the problems of slow process of reasoning and generating natural language text and low overall throughput of LLM. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a text generation method, apparatus, storage medium device, and program product.

[0005] According to a first aspect of one or more embodiments of this specification, a text generation method is proposed, which is applied to a large-scale language model (LLM), including: generating output text through multiple cycles for input text, wherein at least one cycle includes:

[0006] According to the speculative sampling method, a first text sequence after the obtained current text sequence is estimated, and a plurality of candidate sequences are formed according to different subsequences of the current text sequence and the first text sequence;

[0007] In a key-value cache, a logical block is allocated to a text unit in a plurality of candidate sequences for storing attention information of the text unit, and the allocated logical block is mapped to a physical block according to a first criterion; the first criterion includes that in the plurality of candidate sequences, a plurality of logical blocks allocated to the same text unit are mapped to the same physical block;

[0008] The LLM uses the attention information of each candidate sequence in the key-value cache to determine the newly generated text unit in this cycle to form the current text sequence in the next round.

[0009] According to a second aspect of one or more embodiments of the present specification, a text generation device is provided, which is applied to LLM, and generates output text through multiple cycles for input text, wherein at least one cycle is performed by the following units:

[0010] A speculative sampling unit, configured to estimate a first text sequence after the obtained current text sequence according to a speculative sampling method, and form a plurality of candidate sequences according to different subsequences of the current text sequence and the first text sequence;

[0011] A storage space allocation unit, configured to allocate logical blocks to text units in a plurality of candidate sequences in a key-value cache, for storing attention information of the text units, and to map the allocated logical blocks to physical blocks according to a first criterion; the first criterion includes that, in the plurality of candidate sequences, a plurality of logical blocks allocated to the same text unit are mapped to the same physical block;

[0012] The LLM reasoning unit is used for the LLM to use the attention information of each candidate sequence in the key-value cache to determine the newly generated text unit in this cycle to form the current text sequence of the next round.

[0013] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the text generation method as described in the first aspect of the embodiments of this specification is implemented.

[0014] According to a fourth aspect of the embodiments of this specification, a computer device is provided, the computer device comprising:

[0015] processor;

[0016] a memory for storing processor-executable instructions;

[0017] The processor implements the text generation method as described in the first aspect of the embodiment of this specification by running the executable instructions.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, and when the computer program product is executed by a processor, the text generation method as described in the first aspect of the embodiments of this specification is implemented.

[0019] The present specification provides a text generation method, apparatus, storage medium device and program product, which are applied to LLM. In each cycle of LLM, the following can be done: by speculative sampling, the first text sequence after the current text sequence can be obtained, and multiple alternative sequences can be formed. In the key-value cache, logical blocks are allocated to the text units in the multiple alternative sequences, and the logical blocks are mapped to physical blocks. In the process of mapping the physical blocks, mapping is performed according to the first criterion. Specifically, in the multiple alternative sequences obtained by speculative sampling, multiple logical blocks allocated to the same text unit are mapped to the same physical block. The attention information stored in the key-value cache is then used to determine the newly generated text unit in this cycle through LLM.

[0020] Through the above method, firstly, the speculative sampling method is used to increase the number of tokens generated in a cycle, reduce the number of cycles, and speed up the reasoning process of LLM. Secondly, the paging management method is used to store more batches in the KV cache, thereby increasing the number of requests that LLM can process simultaneously and increasing the overall throughput of the system.

[0021] Moreover, in the method provided in this specification, for the special scenario where the same text unit exists in multiple alternative sequences when speculative sampling and paging management are combined, a first criterion for mapping logical blocks to physical blocks is proposed, so that multiple alternative sequences can occupy as little video memory space as possible, further increasing the utilization rate of the video memory space.

[0022] In addition, based on the speculative sampling method used in the above method, in the process of generating output text, the amount of parallel computing in one cycle increases, and the number of cycles decreases. This improves the computing-to-memory ratio, and can still give full play to the hardware parallel capability when the network traffic in online scenarios is limited, making hardware resource utilization higher.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0025] Figure 1 It is a schematic diagram of a key-value cache in the prior art shown in this specification;

[0026] Figure 2 is a flowchart of a text generation method shown in this specification according to an exemplary embodiment;

[0027] Figure 3 It is a partial schematic diagram of a text generation method shown in this specification;

[0028] Figure 4 is another partial schematic diagram of a text generation method shown in this specification;

[0029] Figure 5 is a flowchart of a text generation method shown in this specification according to a specific embodiment;

[0030] Figure 6 is a block diagram of a text generation device according to an exemplary embodiment of the present specification;

[0031] Figure 7It is a hardware structure diagram of a computer device shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0033] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0034] LLM is a neural network model that can generate output text based on user input text. The user's input text can be a question to be answered, a text to be summarized, or a text to be translated. LLM can be used to answer questions, generate text summaries, and translate text.

[0035] The process of LLM generating output text based on input text is generally as follows: first convert the input text into a form that LLM can process, that is, convert it into a sequence of text units (Tokens); then perform reasoning, that is, input the Token sequence of the input text into LLM, encode and represent the token sequence, and predict the output Token sequence based on the encoded representation; finally, convert the output Token sequence into text form to obtain the output text. Token refers to the smallest text unit processed by the LLM model, such as a word or character combination (root) in English, a single Chinese character, and so on.

[0036] The LLM model reasoning process in related technologies has the following problems:

[0037] First, the reasoning process is slow.

[0038] In the above reasoning process of obtaining the output text, the previous order information is needed to infer the current token to be output, so there is a sequential dependency between the generated tokens, and they must be decoded serially. In other words, the process of generating the output text requires multiple cycles, and in each cycle, LLM needs to use the generated tokens and the tokens corresponding to the input text to generate the next token of the output text. This makes the reasoning process of LLM relatively slow.

[0039] Second, the system throughput is low.

[0040] The attention mechanism is required in the LLM processing process to encode and represent the token in combination with the context. The information encoded in this way can be called attention information. In each loop, the next token is generated by the token of the output text that has been generated and the token corresponding to the input text. This makes the attention information used in each loop include all the attention information used in the previous loop. Therefore, in order to reduce the amount of calculation of attention information, a key-value cache (KV cache) is stored in the video memory. The key-value cache stores the attention information of the token used in the process of generating the output text, which can reduce the amount of calculation in the loop process and speed up reasoning.

[0041] For ease of understanding, the concept of request in the LLM processing process is introduced here. LLM can process multiple requests simultaneously in one cycle. Each request corresponds to an input text (usually from the user), and different requests correspond to different input texts. In one cycle, LLM can generate a subsequent token for the text in a request. Figure 1 A batch in corresponds to a request.

[0042] For the specific method of storing attention information in KV cache in related technologies, see Figure 1 The storage space in the KV cache is divided into multiple blocks according to rows and columns. Each row is used to store a batch. A batch corresponds to a request. For requests in different cycles of the same input text, the same batch is generally used to store the attention information required for requests in different cycles.

[0043] Since the output text lengths of each request are different, this will lead to differences in the lengths of different batches. When the lengths of different batches are different, the space size of the KV cache is proportional to the number of batches and the longest sequence length (max_seq_len). In other words, the number of batches that can be stored in the KV cache is limited by the longest batch. Specifically, although the lengths of Batch0, Batch1, and Batch2 are all less than 9 blocks, since the length of Batch3 is 9 blocks, 9 blocks need to be allocated to Batch0, Batch1, and Batch2, and then Figure 1 As shown in the figure, the shaded area is the KVcache space that is actually needed, resulting in the non-shaded area being wasted. If the batch with the longest sequence length is very long, it will be necessary to allocate the same amount of video memory space as the batch with the longest sequence length to all batches, which will result in a large amount of video memory space. Figure 1 The non-shaded part is also wasted. Moreover, when the KV cache size is limited, this will limit the number of batches and prevent them from being set too large, resulting in a small number of requests that can be processed in a sequential cycle by the system, and a low overall throughput.

[0044] In addition, it should be noted that KV cache is based on Figure 1 The storage method shown is to facilitate the calculation in the LLM reasoning process. Specifically, when LLM uses attention information, it is necessary to form a matrix of attention information of multiple requests, and different rows of the matrix store attention information of different requests. The matrix storing the attention information is multiplied by the weight matrix to obtain the output information. Therefore, according to Figure 1 The above matrix can be easily obtained by storing it in Figure 1 Batch0, Batch1, etc. are stored continuously in the Figure 1 Using the KV cache in the image to obtain the matrix that stores attention information will not be able to distinguish different batches and obtain accurate results.

[0045] There are two solutions in the related art for the above two problems. The following will describe the solutions of the two related arts respectively.

[0046] First, the LLM acceleration method based on speculative sampling.

[0047] The speculative sampling process is generally as follows: through a simplified language model, autoregressive sampling is performed on a request to continuously generate multiple tokens, and then the multiple generated tokens are combined with the generated tokens. The multiple tokens obtained by speculative sampling are verified through the LLM large model. If the multiple tokens are considered suitable after verification, the next cycle will continue. If not, the large model will regenerate a token and continue the next cycle.

[0048] Compared with the scheme without speculative sampling, this scheme can increase the number of tokens that can participate in the calculation in one cycle, reduce the number of cycles, accelerate the reasoning process of LLM, and reduce request latency.

[0049] However, this solution cannot solve the problem of low system throughput and the limited number of batches stored in the KV cache. Since the sequence length of the batches that the KV cache can store and the number of batches are mutually restrained, the number of batches that can be processed simultaneously is small when the speculative sampling method may increase the length of the batch, and the system throughput is still low.

[0050] Second, LLM library based on paging management.

[0051] The related art uses the PagedAttention mechanism to solve the problem of low system throughput. Specifically, the KV cache is organized into fixed-size blocks, similar to pages in virtual memory. Logically, according to the following Figure 1 When mapping logical blocks to physical blocks, the logical blocks can be stored continuously or in any free physical blocks, and the mapping relationship between the logical blocks and the physical blocks is recorded in a block table.

[0052] This method improves the utilization rate of physical storage space and reduces the waste of video memory.

[0053] However, this method still fails to accelerate the LLM reasoning process, and the request latency is high. In addition, due to the instantaneous traffic limit of the online scenario network, the number of requests arriving in the same time period is small, which cannot give full play to the advantages of Batch.

[0054] It can be seen that the methods in the related art cannot simultaneously solve the above-mentioned problems of slow LLM reasoning and low system throughput caused by waste of video memory.

[0055] Based on this, this specification proposes a text generation method, which is applied to LLM. In each cycle of LLM, the following can be done: by speculative sampling, the first text sequence after the current text sequence can be obtained, and multiple alternative sequences can be formed. In the key-value cache, logical blocks are allocated to the text units in the multiple alternative sequences, and the logical blocks are mapped to physical blocks. In the process of mapping the physical blocks, mapping is performed according to the first criterion. Specifically, in the multiple alternative sequences obtained by speculative sampling, multiple logical blocks allocated to the same text unit are mapped to the same physical block. The attention information stored in the key-value cache is then used to determine the newly generated text unit in this cycle through LLM.

[0056] Through the above method, firstly, the speculative sampling method is used to increase the number of tokens generated in a cycle, reduce the number of cycles, and speed up the reasoning process of LLM. Secondly, the paging management method is used to store more batches in the KV cache, thereby increasing the number of requests that LLM can process simultaneously and increasing the overall throughput of the system.

[0057] At the same time, in the method that only uses speculative sampling, due to the limitation of KV cache, the length of the batch is limited while the number of batches is guaranteed, so the number of tokens that can be obtained in one round of speculative sampling is small. The combination of speculative sampling and batching removes the limitation on the batch length compared to the above method, and the number of tokens that can be speculatively sampled is larger, which increases the overall throughput. At the same time, the number of cycles is further reduced, which speeds up the inference process and reduces the end-to-end delay.

[0058] Moreover, the above method is not only a direct combination of the paging management and speculative sampling methods, but an improvement based on the combination of the two. Specifically, the method provided in this specification provides a first criterion, and the first criterion is proposed for the needs that only exist in the scenario of combining paging management and speculative sampling. In the scenario of paging management, if speculative sampling is not performed, there will be no alternative sequence composed of multiple subsequences of the first text sequence, then different requests are for different input texts, and there will be no identical text units in different requests, and thus there will be no need to map the logical blocks of the identical text units to the same physical blocks.

[0059] From the above analysis, it can be seen that in the method provided in this specification, for the special scenario where the same text unit exists in multiple alternative sequences when speculative sampling and paging management are combined, a first criterion for mapping logical blocks to physical blocks is proposed, so that multiple alternative sequences can occupy as little video memory space as possible, further increasing the utilization rate of the video memory space.

[0060] In addition, the above method also improves the computation-to-memory access ratio.

[0061] Specifically, although the reasoning process of LLM in the prior art is slow, the slow reasoning is not caused by limited hardware resources, and even the reasoning process does not fully utilize the computing performance of hardware resources. Specifically, due to the limitation of instantaneous network traffic, the number of output texts that can be calculated by the server in one cycle is limited, which leads to a small number of tokens involved in the calculation during one cycle. Moreover, all weight parameters of the LLM model need to be transferred from the storage unit to the calculation unit during each cycle. It can be seen that in one cycle, a lot of time is consumed in memory access operations, and less time is used for calculation; that is, a small number of tokens are calculated in each cycle and the computing performance of hardware resources is not fully utilized, but a lot of time is consumed due to memory access operations.

[0062] Based on the speculative sampling method used in the above solution, in the process of generating output text, the amount of parallel computing in one cycle increases, and the number of cycles decreases. This improves the computing-to-memory ratio, and can still give full play to the hardware parallel capability when the network traffic in online scenarios is limited, making hardware resource utilization higher.

[0063] Next, a text generation method provided in this manual will be described. Figure 2 As shown, Figure 2 is a flowchart of a text generation method according to an exemplary embodiment of the present specification. The method is applied to LLM. In the text generation method, for input text, output text is generated through multiple cycles, wherein at least one cycle includes the following steps: Figure 2 The following steps are shown.

[0064] Step 201 , based on a speculative sampling method, a first text sequence after the obtained current text sequence is estimated, and a plurality of candidate sequences are formed based on different subsequences of the current text sequence and the first text sequence.

[0065] Specifically, in order to speed up the reasoning speed of LLM, a method with simpler complexity is used to estimate multiple text units after the current text sequence, which can form the first text sequence. Thus, multiple candidate sequences can be formed to verify the accuracy of each text unit in the first text sequence.

[0066] Step 201 can be divided into two stages. The first stage is a stage of performing speculative sampling to obtain a first text sequence, and the second stage is a process of forming multiple candidate sequences according to the speculative sampling results.

[0067] Next, we will explain the first stage. First, we will explain the nouns involved in the first stage.

[0068] Among them, the current text sequence obtained may include tokens converted from the user's input text, and tokens corresponding to the output text generated by LLM, or when the first output token has not yet been generated for the input text, it may only include the token sequence of the input text, or when the output token has already been generated, it may only include the generated output text token.

[0069] The current text sequence obtained can correspond to a request, which refers to a request for processing a user input text in a cycle. For the convenience of description, in this article, the request is divided into encoding request and decoding request. The encoding request refers to the request to generate the first token (referred to as the first token) of the output text, and the decoding request refers to the request to generate other tokens in the output text except the first token.

[0070] Correspondingly, when the currently processed request is an encoding request, the currently obtained text sequence may include the tokens converted from the input text. When the currently processed request is a decoding request, the currently obtained text sequence may include the tokens converted from the input text and the tokens corresponding to the output text generated by LLM.

[0071] In addition, the above explanation describes the processing of a request. In the LLM reasoning process, not only can one request be processed at a time, but multiple requests can also be processed simultaneously in a cycle. In other words, the input text can also include multiple user request texts. Correspondingly, the current text sequence includes several sequence segments, each sequence segment corresponds to a request, and includes the user request text, and optionally the generated output text.

[0072] In the case of processing multiple user request texts at the same time, the following step 202 processes the candidate sequences of multiple users in parallel.

[0073] In an optional implementation, if multiple requests are processed simultaneously, only the decoding request may be speculatively sampled, and the encoding request may not be speculatively sampled, and only the attention information of the encoding request may be stored in a KV cache manner. In other words, the input text may include multiple user request texts, and the current text sequence may include several sequence segments, and a single sequence segment corresponds to an output text generated for a single user request text (that is, the current text sequence includes the generated output text); the first text sequence includes several continuation sequences estimated for the several sequence segments.

[0074] The reason why only decoding requests are speculatively sampled is that for encoding requests, although speculative sampling can improve the overall processing speed, it does not improve the first token delay, and the acceleration effect on encoding requests is limited. Therefore, speculative sampling is only performed on decoding requests, and speculative sampling is not performed on encoding requests, which can ensure that the first token delay is not affected. In this way, when users have requirements for the first token delay, the user experience can be guaranteed.

[0075] Of course, when multiple requests are processed simultaneously, speculative sampling may be performed on both encoding requests and decoding requests, and this specification does not limit this.

[0076] After explaining the current text sequence, the first text sequence will be explained next. The first text sequence is a sequence of subsequent text units (Tokens) of the current text sequence obtained by speculative sampling. For example, if the text corresponding to the current text sequence obtained is "Today's weather", the text corresponding to the first text sequence may be "It is sunny".

[0077] In addition, for a request, one result may be obtained by sampling as described above, or multiple results may be obtained by sampling. In the case of obtaining multiple results by sampling, the first text sequence may include multiple possible sequences subsequent to the current text sequence. For example, the text corresponding to the current text sequence is "Today's weather", and the text corresponding to the first text sequence may be two possible results, "It is sunny" and "It is cloudy".

[0078] Both speculative sampling and multiple results have their own advantages. Speculative sampling can reduce the amount of LLM calculations and the repeated writing and returning of KV cache. Speculative sampling can improve the accuracy of the results obtained by speculative sampling, thereby reducing the number of cycles and improving the LLM reasoning acceleration effect of speculative sampling.

[0079] Next, speculative sampling is explained.

[0080] Speculative decoding means predicting a set of results through untrusted methods before performing real generation operations, and then verifying the correctness of the predicted results through a trusted computing process. Generally speaking, the computational cost of untrusted methods is much lower than that of trusted methods.

[0081] The implementation method of speculative sampling can be the same as the implementation method of the related technology introduced above, that is, a simplified language model is used to obtain multiple tokens at one time. In addition, for the implementation method of speculative sampling, speculative sampling can also be implemented based on a vocabulary. In the process of speculative sampling based on the vocabulary, the current text sequence can be used as the key value, and the matching mapping value is queried in the vocabulary as the first text sequence. Of course, the two speculative sampling examples shown in this specification do not represent a limitation on this specification, and other speculative sampling methods can also be used in the method shown in this specification, and this specification does not limit this.

[0082] Next, the method of implementing speculative sampling based on the vocabulary will be explained in detail.

[0083] In an optional implementation, a mapping relationship between a current text sequence and a subsequent text sequence may be stored in a vocabulary, and a possible first text sequence subsequent to the current text sequence may be quickly obtained by matching in the vocabulary.

[0084] In other words, the process of speculative sampling may include: using the current text sequence to query a vocabulary to obtain a matching first text sequence. The vocabulary includes a plurality of mapping relationships, the mapping relationship being a mapping relationship between a text sequence and a subsequent text sequence following the text sequence, and the vocabulary is obtained based on the text generated by the LLM.

[0085] The vocabulary includes mapping relationships, that is, mapping relationships between a text sequence and its subsequent text sequence. For example, mapping relationships such as "tomorrow's weather" - "is sunny", "the weather is" - "sunny" can be stored.

[0086] The mapping relationship in the vocabulary can be obtained based on the text generated by the LLM model in the past.

[0087] In addition, in order to improve the matching degree between the output text and the input text and improve the accuracy of the estimated first text sequence, the vocabulary may include a global vocabulary and a local vocabulary. The global vocabulary caches all high-frequency mapping relationships since the service was started, while the local vocabulary caches the high-frequency mapping relationships generated in the process of generating an output text for a certain input text.

[0088] In other words, the vocabulary includes a global vocabulary and a local vocabulary; the global vocabulary records, in the text generated by LLM, the mapping relationship of the sequence co-occurrence frequency greater than the first frequency threshold; the local vocabulary records, in the text generated for the input text, the mapping relationship of the sequence co-occurrence frequency greater than the second frequency threshold; after the output text is generated, the local vocabulary is added to the global vocabulary.

[0089] The co-occurrence frequency refers to the frequency at which two sequences in the mapping relationship appear simultaneously and successively in the corpus. The first frequency threshold and the second frequency threshold may be the same or different, and the specific value may be selected according to the needs of the scene.

[0090] After the output text is generated, since there is no need to generate text related to the corresponding input text, the local vocabulary can be added to the global vocabulary.

[0091] By adding a local vocabulary, some mapping relationships that are not common in other output texts but are more common in the scenario of the current input text can be added, thereby improving the accuracy of the matching results.

[0092] In addition, when matching, the local word list can be used for matching first to improve the matching degree between the matching result and the current scene. It is also possible to match in both word lists to improve the richness of the result on the basis of improving the matching degree between the matching result and the current scene.

[0093] In addition to storing mapping relationships, the vocabulary can also store the scoring values ​​of the mapping relationships, which can be positively correlated with the above co-occurrence frequency. When performing speculative sampling, if the current text sequence matches multiple text sequences, the text sequence with the highest scoring value can be used as the first text sequence. This can improve the accuracy of the speculative sampling results, reduce the number of tokens involved in the calculation in LLM, and speed up the LLM reasoning process to a certain extent.

[0094] In other words, the mapping relationship in the vocabulary corresponds to a scoring value determined based on the co-occurrence frequency; the method of using the current text sequence to query the vocabulary to obtain a matching first text sequence includes: querying and matching the current text sequence in the vocabulary; when there are multiple matching text sequences in the vocabulary, determining the text sequence with the highest scoring value as the first text sequence.

[0095] In addition, in the presence of scoring values, in order to increase the accuracy of speculative sampling, the scoring values ​​of the mapping relationships that have not been hit in a period of time in the global vocabulary can be reduced, and the reduction size can be determined according to the change in the co-occurrence frequency.

[0096] In another optional implementation, a plurality of sentences may be stored in the vocabulary, and the plurality of sentences may be sentences generated by LLM. When obtaining the first text sequence, the current text sequence may be matched with the plurality of sentences, and when there is text matching the current text sequence in a sentence in the vocabulary, the text sequence following the current text sequence in the sentence may be used as the first text sequence.

[0097] For example, the current text sequence is "tomorrow's weather", and the word list stores the sentence "tomorrow's weather is sunny". In the process of matching the current text sequence with the word list, it is determined that "tomorrow's weather is sunny" matches "tomorrow's weather", and the sentence "tomorrow's weather" following "is sunny" can be used as the first text sequence.

[0098] It should be noted that in the examples above, for ease of understanding, the mapping relationship, text sentences, etc. are given in the form of text. However, in actual application, the vocabulary stores the tokens corresponding to the text. The current text sequence and the first text sequence also exist in the form of token sequences. Similarly, the text mentioned later also exists in the form of tokens, which will not be repeated here.

[0099] Next, some commonalities between the above two vocabularies will be explained.

[0100] When the service application time corresponding to the LLM model is short and the generated text is small, the vocabulary can also be obtained based on the text obtained from the network. In this case, when the service application time corresponding to the LLM model increases and the generated text is sufficient to constitute the vocabulary, the original vocabulary obtained from the text obtained from the network can be deleted to ensure the accuracy of the speculative sampling results. Of course, in order to ensure the diversity of the speculative sampling results, the vocabulary obtained from the text obtained from the network can also be retained.

[0101] If there is no first text sequence matching the current text sequence in the vocabulary, then speculative sampling can be performed on the current text sequence through other speculative sampling methods. Alternatively, the next token of the current text sequence can be directly obtained through LLM.

[0102] In addition, when searching in the vocabulary, all tokens of the current text sequence can be used as key values ​​for searching.

[0103] In some cases, the generated output text is long. If all tokens of the current text sequence are used as key values ​​for query, matching may fail.

[0104] Therefore, in addition to using all tokens of the current text sequence as key values ​​for querying, you can also use the last several tokens of the current text sequence (the number can be preset) as key values ​​for matching query. For example, if the current text sequence is "tomorrow's weather is", you can select "weather is" as the key value for matching query.

[0105] Furthermore, when the vocabulary stores mapping relationships, considering that most characters can be represented by a single token or two tokens, a single token or a token pair can be used as the vocabulary key value. Then the token used for matching in the vocabulary can be the last single token or token pair of the current text sequence, and the key value of the mapping relationship in the vocabulary is also composed of a single token or a token pair.

[0106] In addition, as described above, speculative sampling can be performed on encoding requests. For encoding requests, the current text sequence can be all tokens corresponding to the input text, or the last several tokens of the input text.

[0107] In addition, when LLM processes multiple requests at the same time, multiple requests can be speculatively sampled in one cycle. Correspondingly, the current text sequence includes several sequence segments, and each sequence segment can be matched in the vocabulary, and several consecutive sequences obtained by speculatively sampling the several sequence segments can form the first text sequence.

[0108] After describing the first stage of step 201 , the second stage of step 201 , ie, the process of forming a candidate sequence, will be described next.

[0109] After obtaining the first text sequence, since the LLM model can only output one subsequent token based on the input token at a time, it cannot directly verify the first text sequence. Therefore, in order to facilitate the LLM in the subsequent steps to determine whether each token in the first text sequence is accurate, it is necessary to organize the first text sequence and the current text sequence into the form of alternative sequences. This process can also be called request splitting.

[0110] A single candidate sequence among the multiple candidate sequences includes the current text sequence, or includes a subsequence of the current text sequence and the first text sequence intercepted from the beginning. That is, the subsequence is a continuous subsequence in the first text sequence, and the subsequence includes the first several tokens of a possible speculative sampling result in the first text sequence. Different candidate sequences are different, and different candidate sequences are used to verify different tokens in the first text sequence.

[0111] For example, the current text sequence is "tomorrow's weather", and the first text sequence includes two sequence segments "is sunny" and "is cloudy". Correspondingly, the candidate sequences include: "tomorrow's weather", "tomorrow's weather is", "tomorrow's weather is sunny", "tomorrow's weather is sunny", "tomorrow's weather is cloudy", "tomorrow's weather is cloudy".

[0112] By inputting each candidate sequence into LLM in a subsequent step, LLM can output a subsequent Token for each candidate sequence, and by comparing the Token with the Token at the corresponding position of the first text sequence, multiple Tokens can be output in one cycle.

[0113] Step 202: In the key-value cache, logical blocks are allocated to the text units in the multiple candidate sequences for storing the attention information of the text units, and the allocated logical blocks are mapped to physical blocks according to the first criterion.

[0114] The first criterion includes that, in the multiple candidate sequences, multiple logical blocks allocated to the same text unit are mapped to the same physical block.

[0115] Specifically, the idea of ​​paging management is adopted in step 202, which reduces the number of paging blocks by allocating logical blocks in the key-value cache and then mapping them into physical blocks. Figure 1 At the same time, based on this, speculative sampling will obtain multiple candidate sequences with the same Token. For a user request text in a cycle, the attention information of the same Token in multiple candidate sequences is assigned different logical blocks in the logical unit, but these different logical blocks are mapped to the same physical block. This is convenient for quickly reading attention information through logical blocks and reduces the occupation of physical storage space, making it easier to store more batches in the KV cache, giving full play to the advantages of batches, and improving the overall throughput of the system.

[0116] Next, the specific implementation method of step 202 will be described.

[0117] First, the specific meaning, storage content and function of KV cache are described in detail in the previous article, and will not be repeated here. The storage method of logical blocks and physical blocks in KV cache will be described in detail later.

[0118] Logical block storage and Figure 1 The storage method shown is similar, each row is used to store the attention information corresponding to a request, that is, the attention information of an alternative sequence. The storage logic block is to facilitate the construction of the attention matrix and facilitate the calculation of LLM in the subsequent steps.

[0119] In other words, the logical storage space corresponding to the key-value cache forms a matrix, and the logical blocks in a row of the matrix are used to store text units in an alternative sequence. Correspondingly, the process of allocating logical blocks is: for each alternative sequence, a new row is added to the matrix, and logical blocks are added in the row for the text units in the alternative sequence.

[0120] Specifically, when allocating logical blocks, a new row can be added to the KV cache and the logical blocks can be mapped according to the block size. For example, for encoding requests, logical blocks can be allocated according to the length of the token corresponding to the input text; for decoding requests, the corresponding number of logical blocks can be allocated according to the maximum length after speculative sampling.

[0121] For example, if a request in the encoding phase (i.e., encoding request) has a length of 5 and a block size of 2, then a new row will need to be added to the KVcache, and 3 logical blocks will be allocated to the request in the row. In addition, since the request is an encoding request, no attention information of the request is stored in the KV cache before the request is input. Therefore, after allocating the logical blocks, the allocated 3 logical blocks need to be mapped to physical blocks for storage.

[0122] For another example, a request in the decoding stage (i.e., a decoding request) is split into three requests (corresponding to different candidate sequences). For any of the split requests, a new row can be added to the logical storage space, and the corresponding logical block is allocated to the request in the row. If the request length is 8, then when the block size is 2, 4 logical blocks need to be allocated to the request. Among the 4 logical blocks, 2 logical blocks have the same attention information as the request before the split, and the attention information of the request before the split has been calculated and stored in the KV cache in the previous cycle, so these two logical blocks can be mapped to the corresponding physical blocks that have been stored. Among the 4 logical blocks, the token corresponding to one logical block overlaps with the token in the other split requests, so the logical blocks corresponding to the same token in the two split requests can be mapped to the same physical block. The content stored in the last remaining logical block is different from that of other requests, so a new physical block can be directly mapped to the logical block.

[0123] In addition, multiple blocks in the physical storage space can be stored continuously or in free storage space according to storage rules. In this way, all physical storage space can be used, so that more batches can be processed at the same time, improving system throughput.

[0124] Step 203: The LLM uses the attention information of each candidate sequence in the key-value cache to determine the newly generated text unit in this cycle to form the current text sequence of the next round.

[0125] Specifically, LLM can be used to process multiple candidate sequences at the same time to obtain multiple tokens in one cycle, and the text sequence of the next cycle can be constructed based on the tokens generated in the current cycle until the output text is obtained.

[0126] Although compared to the method in the related art that does not use speculative sampling, the LLM needs to process more requests in one round of loop, that is, the number of alternative sequences is more. However, for the LLM, the parallel computing ability of the hardware makes the LLM not have much more latency when calculating an appropriate number of requests and a small number of requests in one round of loop. For example, the time for the LLM to process 5 requests is about the same as that to process 50 requests. When processing 50 requests compared to processing 5 requests, the latency is not obvious. This enables the full utilization of the parallel computing ability of the hardware in step 203, thereby reducing the number of loop rounds and accelerating the inference process of the LLM.

[0127] Among them, for the specific form of the LLM, this specification does not make any limitations. The method of this specification can be applied to any existing LLM.

[0128] In step 203, the attention information stored in the key-value cache can be utilized to predict a subsequent Token for each alternative sequence, and based on the Tokens predicted by the LLM for each alternative sequence, the newly generated Tokens in this round of loop are determined. In this way, by using the result of speculative sampling as the input, a subsequent Token of the input can be predicted, thereby generating multiple Tokens in one round of loop.

[0129] In other words, determining the newly generated text units in this round of loop includes: the LLM utilizes the attention information of each alternative sequence in the key-value cache to predict the next text unit of each alternative sequence, and the next text units predicted for multiple alternative sequences are sequentially composed into a second text sequence; comparing the second text sequence with the first text sequence, and based on the comparison result, determining the newly generated text units.

[0130] For example, the alternative sequences include: "Tomorrow's weather", "Tomorrow's weather is", "Tomorrow's weather is sunny", "Tomorrow's weather is a sunny day". Then for "Tomorrow's weather", the LLM can generate the Token corresponding to "is", and this Token can verify whether "is" in the first text sequence is accurate; for "Tomorrow's weather is", the LLM can generate the Token corresponding to "sunny"; for "Tomorrow's weather is sunny", the LLM can generate the Token corresponding to "day"; for "Tomorrow's weather is a sunny day", the LLM can generate a subsequent Token, such as the Token corresponding to "please". The corresponding second text sequence is "is sunny please".

[0131] Comparing this second text sequence with the first text sequence, the matching tokens "is sunny" can be used as the verified correct tokens. In this way, the purpose of generating multiple Tokens in one round of loop is achieved.

[0132] In addition, during the LLM verification process, there will be some tokens with speculative sampling errors, and these tokens are allocated logical blocks and physical blocks in step 202. If there are tokens with speculative sampling errors, the logical blocks and physical blocks occupied by these tokens need to be returned to reduce unnecessary resource occupation.

[0133] For the first token in the first text sequence, if the speculative sampling result of the token is wrong, then the subsequent tokens of the token obtained by speculative sampling are also unreliable, and their allocated logical blocks and physical blocks also need to be returned.

[0134] In addition, for a token in the second text sequence, if the token is different from the token at the corresponding position in the first text sequence, in addition to proving that the speculative sampling result of the token at the corresponding position in the first text sequence is wrong and is an erroneous text unit, it can also be shown that the token following the token in the second text sequence is also unreliable. This is because the token following the token is generated by LLM based on the speculative sampling result. In the case of an error in the speculative sampling result, the token following the token in the second text sequence is also unreliable.

[0135] In other words, it is also necessary to: determine the first error text unit based on the comparison result, which is a text unit that exists in the first text sequence but does not exist at the corresponding position in the second text sequence; determine the text unit that follows the first error text unit in the first text sequence as the second error text unit; and for each error text unit in the first error text unit and the second error text unit, return the logical blocks and physical blocks corresponding to the attention information of the error text unit in the key-value cache.

[0136] The first error text unit is obtained by directly comparing the first text sequence with the second text sequence, and the second error text unit is the token following the first error text unit in the first text sequence. The second error text unit may exist in the corresponding positions of the first text sequence and the second text sequence, but it is actually an estimated error text unit.

[0137] In addition, when allocating logical blocks, multiple alternative sequences occupy multiple lines. The multiple lines of alternative sequences in this cycle are no longer needed in the next cycle. Therefore, after determining the newly generated text unit in this cycle, the multiple alternative sequences in the logical block can be merged to facilitate the next cycle. Of course, you can also choose to process the logical blocks occupied by multiple alternative sequences in the previous cycle when allocating logical blocks to the alternative sequences obtained by speculative sampling in the next cycle in the next cycle. This specification does not limit this.

[0138] The above describes how to combine the speculative sampling method based on LLM Batch in the loop process. The following describes when to execute the above method.

[0139] In the process of testing this solution, it was found that without any restrictions on sampling behavior, the token generation speed (Token / s) and throughput (qps) of this solution were increased by 12 times compared to not using this solution, but the corresponding first token delay was also doubled. It can be seen that although the above method can improve the system throughput and the output speed of the output text, the amount of calculation for each cycle in the above process has increased, and the time required for a cycle has increased compared to the related technology. This has led to an increase in the first token delay, which refers to the time consumed from the user sending the input text to the system to the generation of the first token in the output text.

[0140] The requirements for system performance indicators in different scenarios are different. Some scenarios may have high requirements for the first token delay, some scenarios may have high requirements for system throughput, and some scenarios may have high requirements for both the first token delay and system throughput. Therefore, for different scenarios, this specification provides an example that provides three optional modes to support more business scenarios through multiple optional modes. The three optional modes are explained below.

[0141] First, turn off speculative sampling.

[0142] Since speculative sampling increases the amount of computation in a cycle, the amount of data for LLM reasoning increases. At the same time, speculative sampling operations and the allocation and recovery of KV cache will cause additional overhead. Moreover, speculative sampling cannot shorten the generation time of the first Token. Therefore, if speculative sampling is used, it will have a negative impact on the delay of the first Token. Turning off speculative sampling also avoids the negative impact of speculative sampling on the delay of the first Token. In scenarios with high requirements for the delay of the first Token, this mode can be used to avoid an increase in the delay of the first Token.

[0143] Second, all decoding requests are speculatively sampled.

[0144] In other words, when multiple requests are processed in one loop, that is, when the current text sequence includes multiple sequence segments, the process of estimating the first text sequence in the first stage in step 201 includes: performing the speculative sampling only when the LLM has generated at least one output text unit for the multiple user request texts, wherein the current text sequence includes multiple sequence segments corresponding to the multiple user request texts.

[0145] The multiple user request texts have generated at least one output text unit, indicating that the request corresponding to the sequence segment is a decoding request rather than a request to generate the first Token.

[0146] This mode is suitable for applications or services that have high requirements for first-token latency, end-to-end latency (i.e., the latency of generating complete output text), and throughput. Given the observation that speculative sampling has a negative impact on first-token latency and has no acceleration effect on encoding requests, decoding sampling is performed only when a certain input is entirely decoding requests, which can balance multiple performance indicators such as first-token latency, end-to-end latency, and throughput.

[0147] Third, adaptive speculative sampling based on request length.

[0148] First of all, it should be noted that when evaluating the computing power index in the following text, the computing power index is evaluated based on the number of tokens of attention information that need to be calculated in one cycle. The computing power represents the maximum number of tokens that the system can process simultaneously in one cycle without significantly affecting the processing speed. The processing here refers to the processing of calculating attention information. For example, if the computing power is 100 tokens, it means that the system can calculate the attention information of up to 100 tokens in parallel in one cycle without affecting the processing speed.

[0149] It should also be noted that this section is aimed at the scenario of processing multiple requests in one cycle.

[0150] Specifically, when the number of requests processed in a cycle is large, or the number of tokens in the input encoding request is already large, and the hardware performance can be fully utilized, speculative sampling can be stopped, otherwise the response speed may be slowed down. For example, if the hardware computing power indicator is 100 tokens, and the number of tokens in the input encoding request is 100 tokens or even greater than 100 tokens, in this case, there is no excess computing power allocated to the decoding request, so in order not to affect the processing speed too much, speculative sampling can be stopped.

[0151] When the number of input requests is small, or the number of tokens in the input encoding request is insufficient to fully utilize the hardware performance, the computing power consumed by the encoding request can be subtracted from the hardware computing power indicator, and the remaining computing power can be equally divided among multiple decoding requests to determine the number of tokens speculatively sampled for each decoding request, that is, the length of the first text sequence.

[0152] In other words, before speculative sampling, it is also necessary to: determine the hardware computing power indicator and the first number of text units in the target request text; the hardware computing power indicator is used to characterize the maximum number of text units that can simultaneously calculate attention information, and the target request text is the user request text that has not yet generated the first text unit of the output text at the beginning of this cycle; determine the available computing power indicator through the difference between the maximum number of text units and the first number; determine the length of the first text sequence based on the available computing power indicator.

[0153] The target request is also the encoding request. By subtracting the number of target requests from the maximum number of text units, the available computing power index is obtained, which is also the computing power index available for the decoding request. The available computing power index can be allocated to multiple decoding requests according to a preset allocation criterion, thereby determining the length of the first text sequence. The preset allocation criterion can be an average allocation, etc.

[0154] For example, the hardware computing power indicator of a machine is that it can process 100 tokens at the same time, and the total number of tokens in the input encoding request is only 36. Then it can be determined that the available computing power indicator is 64 tokens, and the computing power indicator of 64 tokens can be evenly divided among all decoding requests. Assuming that the number of decoding requests is 16, each decoding request needs to sample 3 tokens speculatively based on the input token.

[0155] Among them, the reason why the first text sequence of speculative sampling for each decoding request in the above example includes 3 tokens instead of 4 tokens is that for the request in the decoding stage, what needs to be calculated in one cycle is not only the attention information of the first text sequence obtained by speculative sampling, but also the tokens whose attention information was not calculated in the previous cycle. Specifically, when the first text sequence estimated by speculative sampling is correct, in step 203, LLM may also generate a subsequent Token for the alternative sequence of the current text sequence plus the first text sequence, and the attention information of this Token needs to be calculated in the next cycle.

[0156] For example, the current text sequence is "tomorrow's weather", and the first text sequence is "it is sunny". The attention information of the token corresponding to "it is sunny" is calculated in this cycle and stored in the KV cache in step 202. For the alternative sequence "tomorrow's weather is sunny", assuming that LLM also generates a token corresponding to "please" in step 203, the attention information of the token corresponding to "please" will be calculated in the next cycle.

[0157] This mode is suitable for applications or services that have high requirements for end-to-end latency and throughput. The above mode can maximize the utilization of computing resources and ensure low end-to-end latency and high throughput while ensuring throughput.

[0158] In addition, the above-mentioned modes can be adjusted according to the performance indicators that the service focuses on to obtain shorter latency or higher throughput. If there are no key performance indicators, the system can also adaptively adjust the sampling length to ultimately obtain a system performance with a relatively balanced latency and throughput.

[0159]

[0160] Table 1

[0161] The performance indicators of the three modes are shown in Table 1. As can be seen from Table 1, when the speculative sampling method is turned off, a better first token delay can be obtained, but the overall request delay and system throughput are poor. When all the requests input in a round of loops are decoding requests, speculative sampling can obtain a slightly better first token delay, and the overall system delay and throughput are slightly better than completely turning off sampling optimization. Adaptive speculative sampling based on the number of requests, sampling sequences of different lengths for batches of different loops according to the number of tokens input in this round of loops, can greatly improve system throughput and shorten request delay, but it will affect the first token delay indicator.

[0162] In summary, compared with the method of simply using a paged KV cache in the related art, the method provided in this specification reduces the number of loops in the text generation process through speculative sampling, thereby reducing the end-to-end delay of requests. At the same time, the method eliminates serial dependence, obtains a first text sequence through speculative sampling, and verifies the first text sequence through LLM, thereby increasing the amount of data in one calculation and making full use of the parallel computing capabilities of the GPU. This solves the problem of not being able to fully utilize the parallel capabilities of the hardware due to the limitation of instantaneous traffic in online scenarios, thereby increasing service throughput.

[0163] Compared with the method that only uses speculative sampling in related technologies, this solution adds paging memory management, which includes the logic block mapping process of the input after sampling and the memory return process after the request is completed, which greatly reduces the memory fragmentation, thereby increasing the number of tokens that can be sampled and further increasing the service throughput. In addition, this manual provides multiple sampling modes, and users can flexibly select the appropriate sampling mode according to the business scenario to obtain better service performance.

[0164] Moreover, the method provided in this specification does not directly superimpose the speculative sampling and paging management methods, but solves the problems that only exist in the superimposed scenarios. This system uses paging memory management to manage KVcache space, divides the KV cache into logical blocks containing specific sequence lengths for allocation, and at the same time, the requests split due to speculative sampling can share the KV cache space with a common prefix, that is, the logical blocks corresponding to the same Token in the alternative sequence are mapped to the same physical block, saving a lot of memory space to carry more Batch requests. This increases the throughput of the service.

[0165] Next, the text generation method provided in this specification will be described through a specific embodiment.

[0166] The service can obtain input text and generate output text based on the input text. This process can be implemented through multiple loops. Each loop can include the following: Figure 3 and Figure 4 The 5 stages are shown. Figure 3 and Figure 4 Together they make up the five stages of each cycle. For ease of reading, Figure 3 and Figure 4 Two figures are used to illustrate the five stages. It is easy to understand that in each cycle Figure 3 After the end of phase 2, you need to execute Figure 4 Stage 3 shown.

[0167] The service consists of multiple modules responsible for executing the loop. The modules are:

[0168] First, the scheduling module: responsible for reading relevant configurations, receiving new requests, and determining whether the request has ended.

[0169] Second, the KV cache management module is responsible for the paging management of the KV cache, and mainly provides functions such as KV cache physical block allocation, recycling, and logical block mapping.

[0170] Third, the LookAhead module is responsible for speculative sampling. This embodiment uses a sampling method based on word list query. It mainly provides functions such as storing token sequences, sampling according to current token information, and extracting the correct sampling sequence.

[0171] Fourth, the Batcher module: mainly responsible for building model input based on request input and speculative sampling information, and building request output data based on model output.

[0172] like Figure 3 and Figure 4As shown, in the process of the service generating multiple output texts for multiple input texts, each cycle includes the following five stages.

[0173] First, at the beginning of the loop, Figure 3 Before the execution of phase 1, the scheduling module first receives new requests and determines whether the requests participating in the previous round of loops have ended, thereby selecting the requests participating in this calculation. And according to the pre-set mode, it determines whether to perform speculative sampling. If speculative sampling is required, it continues to execute Figure 3 and Figure 4 Stages 1-5 are shown. If the speculative sampling module is not required, stage 1 is not executed, and the Batcher module directly builds the model input data, and stage 3 is executed to obtain the token output by this round of cycle.

[0174] In addition, if the adopted mode is adaptive speculative sampling based on request length, the scheduling module can also determine the length of speculative sampling.

[0175] Phase 1, speculative sampling. Performed by the LookAhead module.

[0176] First, for ease of understanding, Figure 4 The meaning of the letters in are explained. Represents the j-th Token of the i-th request. represents the kth Token obtained by speculative sampling for the ith request, k represents its number in the sequence segment of the first text sequence, and j represents its total number in the sequence segments corresponding to the current text sequence and the first text sequence, that is, the total number of tokens processed by LLM for a certain request.

[0177] Figure 3 The input shown in includes 5 requests, among which request 0 and request 1 are encoding requests, and request 2, request 3 and request 4 are decoding requests. Figure 1 The input shown in refers to the length of the sequence in the input vocabulary. Therefore, the tokens of request 0 and request 1 in the encoding phase are counted from 0. Since the request in the decoding phase uses 1 token as the key value when speculative sampling is performed based on the vocabulary, the input requests 2, 3, and 4 only include the last token. The tokens included in each request are detailed in Figure 3 The description in will not be repeated here.

[0178] According to speculative sampling, 3 tokens are sampled from the vocab table for request 2, 1 token is sampled for request 3, and request 4 is not matched in the vocab table and no token is sampled. Figure 3In the figure, the tokens obtained by speculative sampling are indicated by dotted lines.

[0179] For request 2 and request 3, since these two requests obtain multiple tokens through speculative sampling, request 2 and request 3 will be split into multiple requests accordingly.

[0180] Specifically, request 2 is split into R 2 , R 2(1) , R 2(2) and R 2(3) These four requests, R 2 The corresponding candidate sequences include Token: R 2(1) The corresponding candidate sequences include Token: R 2(2) The corresponding candidate sequences include Token: R 2(3) The corresponding candidate sequences include Token: In some implementations, each of the above requests may further include: Request 3 is similar to request 2 and will not be described again here.

[0181] In stage 2, it is necessary to store multiple split requests in different rows of the logical block, and generate a token for each split request in stage 3.

[0182] Phase 2: KV cache memory allocation. This is performed by the KV cache management module.

[0183] First, the KV cache management module can determine whether the logical blocks are sufficient. If not, the physical memory can be allocated first, and then the KV cache logical block mapping can be performed. If the memory blocks are sufficient, the logical block mapping is performed directly, and the physical blocks are allocated to the logical blocks.

[0184] Since logical blocks and physical blocks are not one-to-one corresponding, there are no free logical blocks in the logical storage space, but the physical storage space can still store information. In this case, you can first try to allocate physical blocks for attention information. If you can allocate physical blocks, it means that you can add logical blocks to the logical storage space. Otherwise, if you directly allocate logical blocks, the physical storage space may not be able to store the content corresponding to the logical blocks.

[0185] For each request, you can Figure 3 KV cache video memory is allocated in the manner shown. Figure 3In the figure, the logical block area on the left shows the logical storage space composed of logical blocks, and the physical block area on the right shows the physical storage space. The physical blocks are exemplarily identified and distinguished by numbers 0-35. The number in the logical block means that the logical block will be mapped to the physical block represented by the number. For example, the first logical block in the last row of the logical storage space is marked with the number 3, which means that the logical block is mapped to physical block 3, that is, physical block 3 is used to store the content in the logical block.

[0186] In the boxes corresponding to logical blocks, the boxes shown with dotted lines are the blocks that need to be newly allocated physical blocks this time, and the boxes shown with solid lines are the blocks that have already been allocated physical blocks. In the boxes corresponding to physical blocks, the boxes with dotted lines are the physical blocks that need to be newly mapped in this cycle, the boxes with solid lines are the physical blocks that have already been mapped, and the boxes with gray background are the physical blocks that are occupied after this cycle. Figure 3 The difference in size between the dashed and solid boxes is for ease of reading and does not mean that the contents stored in the two blocks are slightly different. Optionally, the block size can be 2.

[0187] Specifically, for requests 0 and 1 in the encoding phase, since the encoding request is participating in the cycle for the first time, the attention information of these requests has not been stored in the KV cache before. Therefore, it is necessary to allocate logical blocks for the encoding request and map the logical blocks to physical blocks, that is, physical blocks 8-11 are newly allocated and newly mapped.

[0188] For request 4, since speculative sampling does not sample a token, there is no need to allocate new logical blocks and physical blocks for this request in this round of cycles.

[0189] For request 2 and request 3, it is necessary to first determine the multiple candidate sequences obtained after speculative sampling, then add a new row in the logical space for each candidate sequence, and map the logical blocks to physical blocks. When mapping physical blocks, the logical blocks corresponding to the same Token must be mapped to the same physical blocks.

[0190] For request 2, it is split into 4 requests, and each of the 4 requests occupies a row in the logical space for storage. Among them, although the first logical block of each of the 4 requests is newly allocated in the logical storage space for the request split from request 2, it stores The attention information of the Token before it has been calculated in the previous cycle and stored in physical block 0. The subsequent logical block stores the attention information of the Token obtained by speculative sampling, so physical blocks (i.e., blocks 12 and 13) need to be allocated for storage in this cycle.

[0191] The physical block allocation process of request 3 is similar to that of request 2 and will not be described again here.

[0192] Phase 3: LLM reasoning. The Batcher module constructs model input data and performs model reasoning. During the model reasoning process, attention information in the KV cache needs to be read.

[0193] like Figure 4 As shown in Figure 1, the LLM model reasoning process in stage 3 includes: inputting each request (including the request corresponding to the candidate sequence) into the neural network model, and processing it by one or more layers of the neural network model. For example, the input data can be processed by the normalization layer (LayerNorm) first. Then the processed data is input into the attention layer (Attention). The processing formula of the attention layer is shown in Figure 4 . Figure 4 The calculation of attention information takes request 2 as an example. Other requests are similar to request 2 and will not be described in detail. After the attention processing is completed, a series of processing can be performed, such as the normalization layer, to complete the LLM reasoning process.

[0194] Phase 4: Compare the results.

[0195] Specifically, the final result is obtained by performing a misaligned comparison between the LLM model prediction results and the sampled input data.

[0196] Figure 4 The input after speculative sampling shown is the current text sequence and the first text sequence mentioned in step 201, where the Token in the solid box is the current text sequence, and the Token in the dotted box is the first text sequence obtained by speculative sampling. The Token in the dotted box in the LLM generation result is the second text sequence generated by LLM for the alternative sequence. The final result is also the newly generated Token in this round of cycle.

[0197] The input and LLM generation results after speculative sampling are as follows Figure 4 As shown, LLM generates for request 0 This token is generated for request 1. This token. LLM generates a token for request 4. Since these requests do not sample tokens speculatively, there is no need to compare these tokens. These tokens can be directly used as the final results corresponding to these three requests.

[0198] For the alternative sequence corresponding to request 2, for the alternative sequence R 2 Generated This Token, such as Figure 4 As shown in the figure, after the dislocation comparison (i.e., comparing the corresponding tokens in the input after sampling and the generated result in the order of Tokens), it is confirmed that the generated is accurate.2(1) Request generated This Token can confirm different, thus confirming that the first text sequence All tokens after that are inaccurate. Tokens after that are inaccurate, so there is no need to The subsequent tokens are compared with the tokens generated by LLM. At the same time, it can be determined that in the LLM generated results, for R 2(2) Generated And for R 2(3) Generated is inaccurate. Therefore, for request 2, the newly generated token in this cycle includes

[0199] For the candidate sequence corresponding to request 3, the misalignment comparison process is the same as that of request 2, and will not be repeated here.

[0200] After obtaining the final result, the final result can be used as the input of the next cycle, and speculative sampling is performed based on the final result in the next cycle.

[0201] Phase 5: KV cache memory return. Executed by the KV cache management module.

[0202] As mentioned above, through the misalignment comparison in stage 4, it can be determined that there are inaccurate tokens in the first text sequence. The attention information of these tokens does not need to be stored any more, so the logical blocks and physical blocks allocated to these tokens in stage 2 need to be returned.

[0203] Specifically, there are incorrect tokens in the first text sequences corresponding to request 2 and request 3, respectively. Correspondingly, the storage space occupied by the wrong token needs to be deleted. In other words, there are logical blocks that need to be returned in request 2 and request 3. Assuming that the blocks that need to be returned are 13 and 14, then the logical blocks and physical blocks corresponding to 13 and 14 need to be returned.

[0204] In addition, when returning logical blocks and physical blocks, the split requests can be merged into one request and the corresponding rows in the logical block can be deleted. For example, for the multiple rows split from request 2, only the candidate sequence R can be left. 2 This line.

[0205] like Figure 4 As shown, the logical blocks and physical blocks that need to be returned are marked with dotted boxes, and the physical blocks that have been mapped are marked with gray blocks. Figure 4It can be seen that blocks 13 and 14 need to be returned. And in order to facilitate the next cycle, the logic block merges the multiple requests split from request 2 into one request for storage. The same is true for request 3.

[0206] By looping through stage 1 to stage 5, output text can be generated in response to the text input by the user.

[0207] The overall process of the above cycle can be summarized as follows: Figure 5 The flowchart shown includes:

[0208] Step 501: The scheduling module selects requests to participate in this round of calculation.

[0209] Step 502, according to the settings, determine whether to perform speculative sampling. If yes, execute step 503, if not, execute step 504.

[0210] Step 503, perform speculative sampling.

[0211] Step 504, the Batcher module constructs model input data.

[0212] Step 505, determine whether the KV cache memory block is sufficient. If not, execute step 506, and if yes, execute step 507.

[0213] Step 506: KV cache physical block allocation.

[0214] Step 507: KV cache logical block mapping.

[0215] Step 508, LLM model reasoning.

[0216] Step 508: Perform misalignment comparison to obtain the final result.

[0217] Step 509: KV cache memory is returned.

[0218] Figure 5 The specific execution methods of the steps shown are detailed in the previous text and will not be repeated here.

[0219] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus and a computer device to which it is applied.

[0220] like Figure 6 As shown, Figure 6 This is a block diagram of a text generation device according to an exemplary embodiment of the present specification. The device is applied to LLM, and generates output text through multiple cycles for input text, wherein at least one cycle is performed by the following units:

[0221] A speculative sampling unit 610 is used to estimate a first text sequence after the obtained current text sequence according to a speculative sampling method, and form a plurality of candidate sequences according to different subsequences of the current text sequence and the first text sequence;

[0222] A storage space allocation unit 620 is used to allocate logical blocks to text units in multiple candidate sequences in the key-value cache for storing attention information of the text units, and map the allocated logical blocks to physical blocks according to a first criterion; the first criterion includes that multiple logical blocks allocated to the same text unit in the multiple candidate sequences are mapped to the same physical block;

[0223] The LLM reasoning unit 630 is used for the LLM to use the attention information of each candidate sequence in the key-value cache to determine the newly generated text unit in this cycle to form the current text sequence of the next round.

[0224] In an optional implementation, the logical storage space corresponding to the key-value cache forms a matrix, and the logical blocks in a row of the matrix are used to store text units in an alternative sequence; the storage space allocation unit 620 is specifically used to add a new row to the matrix for each alternative sequence, and add logical blocks in the row for the text units in the alternative sequence.

[0225] In an optional implementation, the speculative sampling unit 610 is specifically used to use the current text sequence to query a vocabulary to obtain a matching first text sequence; the vocabulary includes multiple mapping relationships, and the mapping relationship is a mapping relationship between a text sequence and a subsequent text sequence after the text sequence, and the vocabulary is obtained based on the text generated by LLM.

[0226] In an optional embodiment, the vocabulary includes a global vocabulary and a local vocabulary; the global vocabulary records, in the text generated by LLM, the mapping relationship of sequence co-occurrence frequency greater than a first frequency threshold; the local vocabulary records, in the text generated for the input text, the mapping relationship of sequence co-occurrence frequency greater than a second frequency threshold; after the output text is generated, the local vocabulary is added to the global vocabulary.

[0227] In an optional implementation, the mapping relationship in the vocabulary corresponds to a scoring value determined based on the co-occurrence frequency; the speculative sampling unit 610 is specifically used to query and match the current text sequence in the vocabulary; when there are multiple matching text sequences in the vocabulary, the text sequence with the highest scoring value is determined to be the first text sequence.

[0228] In an optional implementation, the LLM reasoning unit 630 is specifically used for the LLM to use the attention information of each alternative sequence in the key-value cache to predict the next text unit of each alternative sequence, and the next text units predicted for multiple alternative sequences are sequentially composed of a second text sequence; the second text sequence is compared with the first text sequence, and the newly generated text unit is determined according to the comparison result.

[0229] In an optional embodiment, it also includes an error text unit determination unit (not shown in the figure), which is specifically used to determine a first error text unit based on the comparison result, which is a text unit that exists in the first text sequence but does not exist at a corresponding position in the second text sequence; determine the text unit that follows the first error text unit in the first text sequence as the second error text unit; and for each error text unit in the first error text unit and the second error text unit, return the logical block and physical block corresponding to the attention information of the error text unit in the key-value cache.

[0230] In an optional embodiment, the input text includes multiple user request texts, the current text sequence includes several sequence segments, and a single sequence segment corresponds to an output text generated for a single user request text; the first text sequence includes several subsequent sequences estimated respectively for the several sequence segments.

[0231] In an optional embodiment, it also includes a length determination unit (not shown in the figure), which is specifically used to determine the computing power index of the hardware and the first number of text units in the target request text; the computing power index of the hardware is used to characterize the maximum number of text units that can simultaneously calculate the attention information, and the target request text is the user request text that has not yet generated the first text unit of the output text at the beginning of this round of cycle; the available computing power index is determined by the difference between the maximum number of text units and the first number; the length of the first text sequence is determined according to the available computing power index.

[0232] In an optional implementation, the speculative sampling unit 610 is specifically configured to perform the speculative sampling only when the LLM has generated at least one output text unit for each of the multiple user request texts, wherein the current text sequence includes multiple sequence segments corresponding to the multiple user request texts.

[0233] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0234] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying creative work.

[0235] like Figure 7 As shown, Figure 7 A hardware structure diagram of a computer device in which a text generation apparatus of an embodiment is located is shown, and the device may include: a processor 1010, a memory 1020 for storing processor executable instructions, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other in communication within the device through the bus 1050. The processor implements the above-mentioned text generation method by running the executable instructions.

[0236] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above method by running executable instructions.

[0237] The memory 1020 for storing processor executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store operating systems and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020.

[0238] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0239] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0240] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0241] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0242] The embodiments of the present specification also provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned text generation method is implemented.

[0243] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0244] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0245] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0246] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0247] The present specification also provides a computer program product, which implements the above-mentioned text generation method when executed by a processor.

Claims

1. A text generation method, applied to a large-scale language model (LLM), comprising: For the input text, the output text is generated through multiple cycles, at least one of which includes: According to the speculative sampling method, a first text sequence after the obtained current text sequence is estimated, and a plurality of candidate sequences are formed according to different subsequences of the current text sequence and the first text sequence; In a key-value cache, a logical block is allocated to a text unit in a plurality of candidate sequences for storing attention information of the text unit, and the allocated logical block is mapped to a physical block according to a first criterion; the first criterion includes that in the plurality of candidate sequences, a plurality of logical blocks allocated to the same text unit are mapped to the same physical block; The LLM uses the attention information of each candidate sequence in the key-value cache to determine the newly generated text unit in this cycle to form the current text sequence in the next round.

2. The method according to claim 1, wherein: The logical storage space corresponding to the key-value cache forms a matrix, and the logical blocks in a row of the matrix are used to store text units in an alternative sequence; The step of allocating logical blocks to text units in a plurality of candidate sequences includes: For each candidate sequence, a new row is added to the matrix, and logic blocks are added to the row for the text units in the candidate sequence.

3. The method according to claim 1, wherein the step of estimating a first text sequence after the current text sequence obtained based on the speculative sampling method comprises: The current text sequence is used to query a vocabulary to obtain a matching first text sequence; the vocabulary includes multiple mapping relationships, the mapping relationship is a mapping relationship between a text sequence and a subsequent text sequence after the text sequence, and the vocabulary is obtained based on the text generated by LLM.

4. The method according to claim 3, wherein: The vocabulary includes a global vocabulary and a local vocabulary; the global vocabulary records, in the text generated by LLM, the mapping relationship of the sequence co-occurrence frequency greater than the first frequency threshold; the local vocabulary records, in the text generated for the input text, the mapping relationship of the sequence co-occurrence frequency greater than the second frequency threshold; after the output text is generated, the local vocabulary is added to the global vocabulary.

5. The method according to claim 3, wherein: The mapping relationships in the vocabulary correspond to scoring values ​​determined based on co-occurrence frequencies; The using the current text sequence to query the vocabulary to obtain a matching first text sequence includes: Performing query matching on the current text sequence in the vocabulary; When there are multiple matching text sequences in the vocabulary, the text sequence with the highest score is determined as the first text sequence.

6. The method according to claim 1, wherein: Determine the text units newly generated in this cycle, including: The LLM uses the attention information of each candidate sequence in the key-value cache to predict the next text unit of each candidate sequence, and the next text units predicted for the multiple candidate sequences are sequentially composed of a second text sequence; The second text sequence is compared with the first text sequence, and a newly generated text unit is determined according to the comparison result.

7. The method according to claim 6, further comprising: Determine a first erroneous text unit according to the comparison result, which is a text unit that exists in the first text sequence but does not exist at a corresponding position in the second text sequence; Determine a text unit in the first text sequence that is subsequent to the first erroneous text unit as a second erroneous text unit; For each erroneous text unit in the first erroneous text unit and the second erroneous text unit, the logical block and the physical block corresponding to the attention information of the erroneous text unit in the key-value cache are returned.

8. The method according to claim 1, wherein: The input text includes multiple user request texts, the current text sequence includes several sequence segments, and a single sequence segment corresponds to an output text generated for a single user request text; the first text sequence includes several subsequent sequences estimated respectively for the several sequence segments.

9. The method according to claim 8, wherein: Before estimating a first text sequence after the obtained current text sequence according to the speculative sampling method, the method further includes: Determine a computing power index of the hardware and a first number of text units in a target request text; the computing power index of the hardware is used to characterize a maximum number of text units that can simultaneously calculate attention information, and the target request text is a user request text that has not yet generated a first text unit of output text at the beginning of this round of loop; Determining an available computing power indicator by the difference between the maximum number of text units and the first number; The length of the first text sequence is determined according to the available computing power indicator.

10. The method according to claim 8, wherein: According to the speculative sampling method, estimating the first text sequence after the current text sequence that has been obtained includes: The speculative sampling is performed only when the LLM has generated at least one output text unit for each of the multiple user request texts, wherein the current text sequence includes multiple sequence segments corresponding to the multiple user request texts.

11. A text generation device, applied to LLM, generates output text through multiple cycles for input text, wherein at least one cycle is performed by the following units: A speculative sampling unit, configured to estimate a first text sequence after the obtained current text sequence according to a speculative sampling method, and form a plurality of candidate sequences according to different subsequences of the current text sequence and the first text sequence; A storage space allocation unit, configured to allocate logical blocks to text units in a plurality of candidate sequences in a key-value cache, for storing attention information of the text units, and to map the allocated logical blocks to physical blocks according to a first criterion; the first criterion includes that, in the plurality of candidate sequences, a plurality of logical blocks allocated to the same text unit are mapped to the same physical block; The LLM reasoning unit is used for the LLM to use the attention information of each candidate sequence in the key-value cache to determine the newly generated text unit in this cycle to form the current text sequence of the next round.

12. A computer device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 10 by running the executable instructions.

13. A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the method according to any one of claims 1 to 10.

14. A computer program product, wherein when the computer program product is executed by a processor, the method according to any one of claims 1 to 10 is implemented.