Method and device for generating text by using large language model

Abstract

The embodiment of the invention provides a method and device for generating a text by using a large language model. The method comprises the steps of obtaining a to-be-matched target sub-sequence from a current lexical element sequence, wherein the current lexical element sequence comprises an input lexical element sequence and a generated lexical element sequence generated by a large language model; the generated lexical element sequence comprises a plurality of special lexical elements used for identifying the beginning of the interval, and the plurality of special lexical elements comprise a first special lexical element closest to the tail of the sequence; from a target word list in a word list library, determining a word element combination matched with the target sub-sequence as a draft word element sequence for speculative decoding; wherein the vocabulary library comprises a plurality of vocabulary corresponding to the plurality of intervals, and the target vocabulary is the vocabulary, corresponding to the interval represented by the first special lexical element, in the plurality of vocabulary; and verifying the draft lexical element sequence by using a large language model to determine a subsequent generated lexical element sequence. And the reasoning speed of the large language model can be effectively improved.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of computers, and more particularly, to a method and apparatus for generating text using a large language model. Background Art

[0002] Large language models (LLMs) are computational models built using deep learning techniques, particularly neural networks, that can understand and generate natural language. These models typically have a large number of parameters, are able to capture complex patterns and contextual relationships in language, and generate natural language text from a given text. They are widely used in natural language processing tasks such as text generation, translation, and question-answering systems.

[0003] In the existing technology, there is a problem of slow inference speed of large language models, and various current acceleration solutions cannot effectively improve the inference speed of large language models. Summary of the Invention

[0004] One or more embodiments of this specification describe a method and apparatus for generating text using a large language model, which can effectively improve the inference speed of the large language model.

[0005] In a first aspect, a method for generating text using a large language model is provided, comprising:

[0006] Obtaining a target subsequence to be matched from a current word-gram sequence, wherein the current word-gram sequence includes an input word-gram sequence and a generated word-gram sequence generated by a large language model; the generated word-gram sequence includes a plurality of special words for marking the beginning of an interval, the plurality of special words including a first special word closest to the end of the sequence;

[0007] Determining, from a target vocabulary in a vocabulary library, a word-gram combination that matches the target subsequence as a draft word-gram sequence for speculative decoding; wherein the vocabulary library includes a plurality of vocabulary tables corresponding to a plurality of intervals, and the target vocabulary table is a vocabulary table in the plurality of vocabulary tables corresponding to the interval represented by the first special word-gram;

[0008] The draft word-gram sequence is verified using the large language model to determine a subsequent generated word-gram sequence.

[0009] In a possible implementation, obtaining a target subsequence to be matched from the current word sequence includes:

[0010] Sequentially concatenate the input word-gram sequence and the generated word-gram sequence generated by the large language model to obtain a prefix word-gram sequence;

[0011] A first predetermined number of word-grams that do not belong to special word-grams are intercepted from the prefix word-gram sequence in order from back to front as target subsequences to be matched.

[0012] In one possible implementation, determining, from a target vocabulary in a vocabulary library, a word-gram combination matching the target subsequence as a draft word-gram sequence for speculative decoding includes:

[0013] Searching for a target term containing the target subsequence from each term included in the target vocabulary in descending order of frequency of occurrence;

[0014] A word-gram combination of a second predetermined number of word-grams is intercepted from the end position of the target subsequence in the target word entry and is used as a draft word-gram sequence.

[0015] In one possible implementation, the verifying the draft word-gram sequence using the large language model to determine a subsequent generated word-gram sequence includes:

[0016] Generate multiple candidate sequences based on the current word sequence and the draft word sequence; wherein a single candidate sequence includes the current word sequence, or includes a subsequence of the current word sequence and the draft word sequence taken from the beginning;

[0017] Each candidate sequence is input into the large language model, and the large language model outputs a subsequent predicted word element for each candidate sequence;

[0018] By comparing the predicted word with the word at the corresponding position of the draft word sequence, it is verified whether the word at the corresponding position of the draft word sequence is correct.

[0019] Furthermore, the method further comprises:

[0020] retaining each word before the first word verified as erroneous in the draft word sequence, and adding the retained word and the predicted word to the generated word sequence to obtain an updated generated word sequence;

[0021] If the updated generated word-gram sequence does not include the second special word-gram representing the end of the interval, the iterative execution determines the target subsequence to be matched from the current word-gram sequence; if the updated generated word-gram sequence includes the second special word-gram, the large language model is triggered to predict the next word-gram.

[0022] Furthermore, the method further comprises:

[0023] If the updated generated word-gram sequence includes the second special word-gram, the word entries consisting of the word-grams in the interval are added to the target word list.

[0024] Furthermore, the method further comprises:

[0025] In response to the number of entries or the storage amount in the target vocabulary reaching a preset threshold, several entries with the lowest usage frequency are deleted.

[0026] In one possible implementation, the multiple vocabularies are pre-generated in the following manner:

[0027] Initializing N cache spaces for storing the multiple word lists;

[0028] For the knowledge text segments in the local knowledge base, the vocabulary corresponding to the interval is added according to the corresponding interval mark.

[0029] Furthermore, the knowledge text segments in the local knowledge base are generated by the large language model, and contain the special word-grams as the interval markers.

[0030] In a second aspect, a device for generating text using a large language model is provided, comprising:

[0031] an acquisition unit, configured to acquire a target subsequence to be matched from a current word-gram sequence, wherein the current word-gram sequence includes an input word-gram sequence and a generated word-gram sequence generated by a large language model; the generated word-gram sequence includes a plurality of special words for marking the beginning of an interval, wherein the plurality of special words includes a first special word closest to the end of the sequence;

[0032] a matching unit, configured to determine, from a target vocabulary in a vocabulary library, a word-gram combination that matches the target subsequence obtained by the obtaining unit as a draft word-gram sequence for speculative decoding; wherein the vocabulary library includes a plurality of vocabulary tables corresponding to a plurality of intervals, and the target vocabulary table is a vocabulary table in the plurality of vocabulary tables corresponding to the interval represented by the first special word-gram;

[0033] A verification unit is used to verify the draft word-gram sequence obtained by the matching unit using the large language model to determine a subsequent generated word-gram sequence.

[0034] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method according to the first aspect.

[0035] In a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0036] Through the method and device provided in the embodiments of this specification, a speculative decoding algorithm is adopted to use a lighter draft model to predict multiple generated word units, and then the large language model is used to batch verify the correctness of the results, thereby speeding up the generation speed of the decoding stage. While ensuring the quality of the model output, the reasoning speed of the autoregressive large language model is fully optimized. On this basis, a multi-word table search and matching method is used to obtain the draft word unit sequence, which greatly reduces the amount of calculation and reduces the need for memory access. The implementation is simple and faster; and by dynamically matching different word tables in different generation intervals, the mutual disturbance between word frequencies in different generation intervals is reduced, the speculative decoding hit rate is improved, the amount of calculation is reduced, and a faster reasoning speed is achieved. The combination of the above-mentioned multiple means can effectively improve the reasoning speed of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;

[0039] Figure 2 A schematic diagram of a speculative decoding process according to one embodiment is shown;

[0040] Figure 3 A flowchart of a method for generating text using a large language model according to one embodiment is shown;

[0041] Figure 4 A schematic diagram of a large language model decoding process according to one embodiment is shown;

[0042] Figure 5 A schematic diagram of partition segment prediction according to one embodiment is shown;

[0043] Figure 6 A schematic block diagram shows an apparatus for generating text using a large language model according to one embodiment. DETAILED DESCRIPTION

[0044] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0045] Figure 1This is a schematic diagram of an implementation scenario of an embodiment disclosed in this specification. This implementation scenario involves generating text using a large language model. It can be understood that a large language model usually generates output text based on input text. A large language model (LLM) is a neural network model that can obtain output text based on user input text. The user's input text can be a question to be answered, a text that needs to generate a summary, or a text that needs to be translated. LLM can be used to achieve tasks such as question answering, generating text summaries, and text translation. The process of LLM generating output text based on input text is generally as follows: first, the input text is converted into a form that can be processed by LLM, that is, converted into a sequence of tokens; then reasoning is performed, that is, the input token sequence converted from the input text is input into LLM, the input token sequence is encoded and represented, and a token sequence is generated based on the encoded representation; finally, the generated token sequence is converted into text form to obtain output text. Among them, token, in the field of natural language processing, refers to the smallest semantic unit in the text, which can be a word, subword, character, etc. The large language model usually divides the input text into several tokens for processing and output generation.

[0046] Reference Figure 1 , i represents the input word sequence, t0 represents the encoded representation corresponding to the input word sequence, and t1, t2, t3, t4, and t5 represent the generated word sequence. LLM reasoning typically requires the use of autoregressive sampling. However, the autoregressive sampling process is quite slow and requires serial decoding of each word. When generating each word, all parameters need to be transferred from the storage unit to the computing unit. Due to the large parameter size of the LLM, the large amount of data transfer increases the burden on memory access bandwidth, so the LLM reasoning process is often limited by memory access speed. In the reasoning process of obtaining the generated word sequence mentioned above, the previous order information is needed to infer the current word to be output. Therefore, there is a sequential dependency between the generated word elements, and serial decoding is required. In other words, the process of generating the output text requires multiple cycles, and in each cycle, the LLM needs to use the generated word elements and the input word sequence corresponding to the input text to generate the next word element of the output text. This results in a relatively slow LLM reasoning process.

[0047] Speculative decoding is a technique used for inference on large language models, designed to improve the efficiency of text generation using these models. This method first generates several candidate tokens by running a smaller, more efficient draft model. These candidate tokens are then verified in parallel using a larger, more accurate large language model. The more accurate candidate tokens are used as the final generated tokens, achieving the effect of generating multiple tokens in a single iteration. This approach significantly reduces computing resource consumption and response time while maintaining generation quality, improving overall generation efficiency.

[0048] Figure 2 FIG. 1 shows a schematic diagram of a speculative decoding process according to an embodiment. Figure 2 , the text generation using speculative decoding mainly includes the following process: first, the draft model Mq generates k words, for example, using the draft model autoregression to generate k words and probability distribution Q(x); then the large language model Mp is forward propagated in parallel, for example, using the large language model Mp to evaluate the words generated by Mq in the previous step in parallel to obtain the probability distribution P(x); then verify and evaluate the results, for example, reject the generation of each word with a probability of 1-P(x) / Q(x), and resample a word from a new probability distribution P′(x)=norm(max(0,P(x)-Q(x))); then return the generated result; repeat the above steps until the generation is completed.

[0049] Speculative decoding aims to address the slow autoregressive sampling and inference issues in large language models. Its core idea is to simultaneously verify multiple draft tokens during a single forward pass. Therefore, the core of this technology lies in generating draft tokens as quickly and accurately as possible. Currently, some mainstream draft models use methods that generate draft tokens slowly, while others have low accuracy and a low probability of acceptance. Therefore, speculative decoding not only fails to improve performance but also continues to waste computing resources and increase request response times.

[0050] In the embodiment of this specification, a vocabulary matching method is adopted to make the process of generating draft word units faster. In addition, in view of the locality of the large language model generation stage in actual application scenarios, that is, different word unit distribution characteristics are usually exhibited in different generation intervals, the design of multiple vocabulary tables is adopted to improve the accuracy of generating draft word units. For example, for the output text "[Arrangement S][1_R][2][3][Arrangement E] Hello, I am your smart assistant Ma Xiaocai", the output text can be divided into two intervals. The first interval is the output content between "[Arrangement S]" and "[Arrangement E]", which only contains the card serial number to be used, and the second interval is the actual answer content. If the two intervals are corresponding to a vocabulary table, the generated content at different stages will interfere with each other. Therefore, a vocabulary table is established for each interval to adapt to the large language model having different word frequency hotspot characteristics in different intervals, thereby improving the accuracy of generating draft word units. Accordingly, the reasoning speed of the large language model can be effectively improved.

[0051] Figure 3 A flowchart of a method for generating text using a large language model according to an embodiment is shown. The method can be based on Figure 1 The implementation scenario shown and Figure 2 The speculative decoding process shown in Figure 3As shown, the method for generating text using a large language model in this embodiment includes the following steps: Step 31, obtaining a target subsequence to be matched from the current word sequence, wherein the current word sequence includes an input word sequence and a generated word sequence generated by the large language model; the generated word sequence includes several special words for identifying the beginning of an interval, and the several special words include the first special word closest to the end of the sequence; Step 32, determining a word combination matching the target subsequence from a target word list in a word list library as a draft word list sequence for speculative decoding; wherein the word list library includes multiple word lists corresponding to multiple intervals, and the target word list is a word list in the multiple word lists corresponding to the interval represented by the first special word list; Step 33, verifying the draft word list sequence using the large language model to determine the subsequent generated word list sequence. The specific execution method of each of the above steps is described below.

[0052] First, in step 31, a target subsequence to be matched is obtained from the current word-gram sequence, where the current word-gram sequence includes the input word-gram sequence and a generated word-gram sequence generated by the large language model; the generated word-gram sequence includes several special words used to mark the beginning of a segment, including the first special word closest to the end of the sequence. It is understood that the target subsequence may be a portion of the current word-gram sequence.

[0053] The current word-gram sequence may include an input word-gram sequence converted from the user's input text and a generated word-gram sequence generated by the LLM, or may only include the input word-gram sequence when the first word-gram has not yet been generated.

[0054] The current token sequence obtained can correspond to a request, which refers to a request for processing a piece of user input text. Specifically, requests can be divided into encoding requests and decoding requests. Encoding requests refer to requests for generating the first token of the output text (referred to as the first token for short), while decoding requests refer to requests for generating tokens other than the first token in the output text.

[0055] Correspondingly, if the currently processed request is an encoding request, the current word sequence obtained may include tokens converted from the input text. If the currently processed request is a decoding request, the current text sequence obtained may include tokens converted from the input text and tokens corresponding to the output text generated by LLM.

[0056] Speculative sampling is performed only on decoding requests because, while speculative sampling can improve overall processing speed for encoding requests, it doesn't improve first-token latency, and its acceleration effect on encoding requests is limited. Therefore, speculative sampling is performed only on decoding requests, while not on encoding requests, to ensure that first-token latency is not affected. This ensures a good user experience even when users have high requirements for first-token latency.

[0057] In one example, obtaining a target subsequence to be matched from the current word sequence includes:

[0058] Sequentially concatenate the input word-gram sequence and the generated word-gram sequence generated by the large language model to obtain a prefix word-gram sequence;

[0059] A first predetermined number of word-grams that do not belong to special word-grams are intercepted from the prefix word-gram sequence in order from back to front as target subsequences to be matched.

[0060] In this example, the first predetermined number can be set as needed according to business conditions, for example, set to 3 according to empirical values.

[0061] Then, in step 32, a word-gram combination matching the target subsequence is determined from a target word list in a word list library as a draft word-gram sequence for speculative decoding. The word list library includes multiple word lists corresponding to multiple intervals, and the target word list is the word list in the multiple word lists corresponding to the interval represented by the first special word-gram. It will be appreciated that the corresponding word list can be determined based on the first special word-gram in the generated word-gram sequence, so that different word lists are matched to different intervals.

[0062] Speculative sampling involves predicting a set of results using untrusted methods before performing actual generative operations. The correctness of the predicted results is then verified using a trusted computational process. Generally speaking, the computational cost of untrusted methods is much lower than that of trusted methods.

[0063] Regarding the implementation of speculative sampling, the embodiment of this specification implements speculative sampling based on a vocabulary. In the process of speculative sampling based on the vocabulary, the target subsequence can be used as a key value, and the matching mapping value is queried in the vocabulary as a draft word sequence.

[0064] In one example, determining, from a target vocabulary in a vocabulary library, a word-gram combination matching the target subsequence as a draft word-gram sequence for speculative decoding includes:

[0065] Searching for a target term containing the target subsequence from each term included in the target vocabulary in descending order of frequency of occurrence;

[0066] A word-gram combination of a second predetermined number of word-grams is intercepted from the end position of the target subsequence in the target word entry and is used as a draft word-gram sequence.

[0067] In this example, the second predetermined number can be set as needed according to business conditions and is compatible with the first predetermined number. For example, the first predetermined number is set to 3 according to empirical values, and the second predetermined number is set to 7 according to empirical values.

[0068] For example, a vocabulary can store multiple sentences, each sentence being a term, and the multiple sentences can be sentences generated by LLM. When obtaining a draft word gram sequence, the target subsequence can be matched with multiple sentences. When there is text matching the target subsequence in a sentence in the vocabulary, the word gram sequence following the target subsequence in the sentence can be used as the draft word gram sequence. For example, the target subsequence is "tomorrow's weather", and the vocabulary stores the sentence "tomorrow's weather is sunny". In the process of matching the target subsequence with the vocabulary, it is determined that "tomorrow's weather is sunny" matches "tomorrow's weather", and the "is sunny" following "tomorrow's weather" in the sentence can be used as the draft word gram sequence.

[0069] It should be noted that the examples above use text to illustrate mapping relationships and sentences for ease of understanding. However, in actual applications, the vocabulary stores tokens corresponding to the text. The target subsequence and draft word sequence also exist as token sequences.

[0070] Finally, in step 33, the draft word-gram sequence is verified using the large language model to determine the subsequent generated word-gram sequence. It is understood that each word-gram in the draft word-gram sequence needs to be verified by the large language model, and it is possible that some word-grams will be verified as correct while others will be verified as incorrect.

[0071] In one example, the verifying the draft word-gram sequence using the large language model to determine a subsequent generated word-gram sequence includes:

[0072] Generate multiple candidate sequences based on the current word sequence and the draft word sequence; wherein a single candidate sequence includes the current word sequence, or includes a subsequence of the current word sequence and the draft word sequence taken from the beginning;

[0073] Each candidate sequence is input into the large language model, and the large language model outputs a subsequent predicted word element for each candidate sequence;

[0074] By comparing the predicted word with the word at the corresponding position of the draft word sequence, it is verified whether the word at the corresponding position of the draft word sequence is correct.

[0075] In this example, the draft word sequence includes k tokens. The draft word sequence is concatenated with the prefix, and k+1 tokens are predicted by the large language model. The correctness of the tokens is verified and evaluated by comparing them with the words in the corresponding positions.

[0076] Furthermore, the method further comprises:

[0077] retaining each word before the first word verified as erroneous in the draft word sequence, and adding the retained word and the predicted word to the generated word sequence to obtain an updated generated word sequence;

[0078] If the updated generated word-gram sequence does not include the second special word-gram representing the end of the interval, the iterative execution determines the target subsequence to be matched from the current word-gram sequence; if the updated generated word-gram sequence includes the second special word-gram, the large language model is triggered to predict the next word-gram.

[0079] In this example, if the updated generated word-gram sequence does not include the second special word-gram representing the end of the interval, it means that the subsequent prediction is still for the same interval, and the same matching method is used subsequently; if the updated generated word-gram sequence includes the second special word-gram, it means that the prediction of the interval is over, and the large language model is triggered to predict the next word-gram. In other words, the prediction of the next word-gram is performed directly by the large language model without the need to generate a draft word-gram sequence.

[0080] Furthermore, the method further comprises:

[0081] If the updated generated word-gram sequence includes the second special word-gram, the word entries consisting of the word-grams in the interval are added to the target word list.

[0082] In this example, the vocabulary is continuously updated, and the above addition may specifically be updating the occurrence frequency of existing entries in the vocabulary.

[0083] Furthermore, the method further comprises:

[0084] In response to the number of entries or the storage amount in the target vocabulary reaching a preset threshold, several entries with the lowest usage frequency are deleted.

[0085] In this example, the frequency of use can be reflected by the frequency of occurrence of the terms, that is, the terms with the lowest frequency of occurrence are deleted. Specifically, the least frequently used (LFU) algorithm can be used. It is a cache replacement algorithm used to manage cache resources in computer memory. Its basic principle is to eliminate the data that is used least frequently. When the cache space is full and old data needs to be removed to make room for new data, the LFU algorithm will select the data with the least frequency of use to replace it.

[0086] In addition, the global vocabulary storage structure can be implemented using a least recently used (LRU) algorithm. When the vocabulary space is full, the least recently used text is selected for elimination. The update time of the entry can be used to determine whether it has been used the least recently. However, this method is susceptible to the influence of short-term cold data and may cause hot data to be eliminated prematurely.

[0087] In one example, the multiple vocabularies are pre-generated in the following manner:

[0088] Initializing N cache spaces for storing the multiple word lists;

[0089] For the knowledge text segments in the local knowledge base, the vocabulary corresponding to the interval is added according to the corresponding interval mark.

[0090] In this example, a corresponding number of vocabularies can be constructed based on the number of intervals, and each vocabularies stores several entries. The above-mentioned knowledge text segments can be derived from the historical conversation data between the user and the large language model, can be the text generated by the large language model, or can be the user's input text.

[0091] Furthermore, the knowledge text segments in the local knowledge base are generated by the large language model, and contain the special word-grams as the interval markers.

[0092] In this example, based on special words, entries can be selected from the knowledge text segment generated by the large language model and added to the corresponding vocabulary.

[0093] Through the method provided in the embodiments of this specification, a speculative decoding algorithm is adopted to use a lighter draft model to predict multiple generated word units, and then the large language model is used to batch verify the correctness of the results, thereby speeding up the generation speed of the decoding stage. While ensuring the quality of the model output, the reasoning speed of the autoregressive large language model is fully optimized. On this basis, a multi-word table search and matching method is used to obtain the draft word unit sequence, which greatly reduces the amount of calculation and reduces the need for memory access. The implementation method is simple and the speed is faster; and by dynamically matching different word tables in different generation intervals, the mutual disturbance between word frequencies in different generation intervals is reduced, the speculative decoding hit rate is improved, the amount of calculation is reduced, and a faster reasoning speed is achieved. The combination of the above-mentioned multiple means can effectively improve the reasoning speed of the large language model.

[0094] Figure 4 FIG. 1 is a schematic diagram showing a large language model decoding process according to an embodiment. Figure 4 The embodiments of this specification propose to perform large language model decoding based on segmented independent prediction, which can improve the hit rate of speculative decoding and greatly reduce the time consumption of the inference stage. The large language model decoding process mainly includes:

[0095] First, the segment probability prediction device is initialized in step 401. It can be understood that the initialization process is a process of initially establishing multiple vocabulary tables.

[0096] The initialization process consists of two sub-steps:

[0097] Step 4011: Initialize the segmented vocabulary. Specifically, initialize N LFU caches to store the vocabulary to be matched in each generation phase. Each LFU cache has a maximum storage capacity of M texts, which can be sentences in the LLM generation process or content in the input text.

[0098] Step 4012: Preheat the segmented vocabulary. Preheat the segmented vocabulary based on the knowledge retrieved from the local knowledge base. Each text item is found in the vocabulary item to be stored through the segmented vocabulary selector. The initial frequency corresponding to each text item is set to F.

[0099] Then, in step 402, the draft token is matched. It can be understood that this step is to obtain the draft token sequence by matching the target subsequence with the vocabulary.

[0100] Among them, the current word list to be selected is obtained through the segmented word list selector, and then the word list to be matched is obtained, and a draft word element sequence with a length of k is obtained from the word list to be matched according to the target subsequence.

[0101] Next, in step 403, the target model is inferred and verified. It can be understood that the target model is a large language model.

[0102] In the embodiment of this specification, the draft word sequence can be concatenated with the prefix, and k+1 tokens can be predicted by the target model. Then, the tokens are compared with the words at the corresponding positions in the draft word sequence to verify whether each word in the draft word sequence is correct.

[0103] Then, the generated result is returned in step 404. It can be understood that the generated result is a generated word sequence obtained according to the verification result.

[0104] Among them, if the generated result contains the end token of the current stage, it is updated to the corresponding LFU vocabulary through the segment vocabulary selector.

[0105] Finally, in step 405, it is determined whether the generation is completed. If the determination result is no, steps 402 to 405 are repeated.

[0106] Specifically, the above-mentioned end of generation may refer to whether the prediction of the interval in the segmented prediction is ended, or may refer to the end of generation of the entire output text for a request.

[0107] Figure 5 FIG. 4 shows a schematic diagram of partition segment prediction according to an embodiment. Figure 5 The process of generating text using a large language model is divided into N intervals, and each interval adds a special word as a marker, where S i represents the starting word of the ith interval, E i Represents the end word of the i-th interval. T0 to T3 represent the input word sequence of the current request, S0 represents the first word generated by the request, and the segmented word list selector is used to determine that the current stage is Stage 0, and the speculative decoding algorithm is based on word list 0. When the end word E0 of this stage is generated, the word generated in this interval and the corresponding frequency are updated to word list 0, and so on, until the generation process ends. Furthermore, the segmented word list selector stores the mapping relationship between special words and generation stages, and can quickly obtain the generation stage corresponding to the current special word. In the process of speculative decoding based on the word list, the sequence to be matched can include the input word sequence of the current request, or the generated word sequence generated by the large language model. The sequence to be matched can be used as a key value, and the matching mapping value can be queried in the word list as a draft word sequence. The length of the sequence to be matched and the length of the draft word sequence can be set to 3 and 7 respectively according to the empirical value, or can be set as needed according to the business situation.

[0108] It should be noted that the partition segment prediction is a prediction for the interval represented by the generated starting special word. If only the word sequence is currently input, the partition segment prediction will not be performed. Only the word will be generated by the large language model, and the partition segment prediction will not be performed until the special word is generated. If the prediction of an interval is completed, the partition segment prediction will not be performed until the special word is generated. For example, T7 in the figure does not have the starting special word and the ending special word, and it is only generated by the large language model.

[0109] According to another embodiment, a device for generating text using a large language model is also provided, and the device is used to execute the method provided in the embodiment of this specification. Figure 6 FIG. 1 is a schematic block diagram of an apparatus for generating text using a large language model according to an embodiment. Figure 6 As shown, the apparatus 600 includes:

[0110] an acquisition unit 61 configured to acquire a target subsequence to be matched from a current word-gram sequence, wherein the current word-gram sequence includes an input word-gram sequence and a generated word-gram sequence generated by a large language model; the generated word-gram sequence includes a plurality of special words for marking the beginning of an interval, wherein the plurality of special words includes a first special word closest to the end of the sequence;

[0111] a matching unit 62 configured to determine, from a target vocabulary in a vocabulary library, a word-gram combination that matches the target subsequence obtained by the obtaining unit 61 as a draft word-gram sequence for speculative decoding; wherein the vocabulary library includes a plurality of vocabulary tables corresponding to a plurality of intervals, and the target vocabulary table is a vocabulary table in the plurality of vocabulary tables corresponding to the interval represented by the first special word-gram;

[0112] The verification unit 63 is configured to verify the draft word-gram sequence obtained by the matching unit 62 using the large language model to determine a subsequent generated word-gram sequence.

[0113] Optionally, as an embodiment, the acquiring unit 61 includes:

[0114] a concatenation subunit, configured to sequentially concatenate the input word-gram sequence and the generated word-gram sequence generated by the large language model to obtain a prefix word-gram sequence;

[0115] The first interception sub-unit is used to intercept a first predetermined number of word-grams that do not belong to special word-grams from the prefix word-gram sequence obtained by the splicing sub-unit in a backward order as a target sub-sequence to be matched.

[0116] Optionally, as an embodiment, the matching unit 62 includes:

[0117] a search subunit, configured to search for a target entry containing the target subsequence from among the entries in the target vocabulary in descending order of frequency of occurrence;

[0118] The second interception subunit is used to intercept a word-unit combination of a second predetermined number of word-units from the end position of the target subsequence in the target word entry obtained by the search subunit, and use it as a draft word-unit sequence.

[0119] Optionally, as an embodiment, the verification unit 63 includes:

[0120] A generating sub-unit, configured to generate a plurality of candidate sequences based on the current word sequence and the draft word sequence; wherein a single candidate sequence includes the current word sequence, or includes a subsequence of the current word sequence and the draft word sequence taken from the beginning;

[0121] The prediction subunit is used to input each candidate sequence obtained by the generation subunit into the large language model, and output a subsequent predicted word element for each candidate sequence through the large language model;

[0122] The verification subunit is used to verify whether the word at the corresponding position of the draft word sequence is correct by comparing the predicted word obtained by the prediction subunit with the word at the corresponding position of the draft word sequence.

[0123] Furthermore, the device further comprises:

[0124] an updating unit, configured to retain each word-gram preceding the first word-gram verified as erroneous in the draft word-gram sequence, and add the retained word-grams and the predicted word-grams to the generated word-gram sequence to obtain an updated generated word-gram sequence;

[0125] A triggering unit is configured to iteratively determine a target subsequence to be matched from the current word-gram sequence if the generated word-gram sequence updated by the updating unit does not include the second special word-gram representing the end of the interval; and trigger the large language model to predict the next word-gram if the generated word-gram sequence updated by the updating unit includes the second special word-gram.

[0126] Furthermore, the device further comprises:

[0127] The vocabulary adding unit is configured to add the entry consisting of each word-gram in the interval to the target vocabulary if the updated generated word-gram sequence includes the second special word-gram.

[0128] Furthermore, the device further comprises:

[0129] The deleting unit is configured to delete a number of entries with the lowest usage frequency in response to the number of entries or the storage amount in the target vocabulary reaching a preset threshold.

[0130] Optionally, as an embodiment, the multiple vocabularies are pre-generated in the following manner:

[0131] Initializing N cache spaces for storing the multiple word lists;

[0132] For the knowledge text segments in the local knowledge base, the vocabulary corresponding to the interval is added according to the corresponding interval mark.

[0133] Furthermore, the knowledge text segments in the local knowledge base are generated by the large language model, and contain the special word-grams as the interval markers.

[0134] The device provided by the embodiment of this specification adopts a speculative decoding algorithm to use a lighter draft model to predict multiple generated word units, and then the large language model is used to batch verify the correctness of the results, thereby speeding up the generation speed of the decoding stage. While ensuring the quality of the model output, it fully optimizes the reasoning speed of the autoregressive large language model. On this basis, a multi-word table search and matching method is used to obtain the draft word unit sequence, which greatly reduces the amount of calculation and reduces the need for memory access. The implementation method is simple and the speed is faster. Moreover, by dynamically matching different word tables in different generation intervals, the mutual disturbance between word frequencies in different generation intervals is reduced, the speculative decoding hit rate is improved, the amount of calculation is reduced, and a faster reasoning speed is achieved. The combination of the above-mentioned multiple means can effectively improve the reasoning speed of the large language model.

[0135] According to another embodiment, there is also provided a computer readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 3 The method described.

[0136] According to another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the system realizes the combination of Figure 3 The method described.

[0137] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0138] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating text using a large language model, comprising: Obtaining a target subsequence to be matched from a current word-gram sequence, wherein the current word-gram sequence includes an input word-gram sequence and a generated word-gram sequence generated by a large language model; the generated word-gram sequence includes a plurality of special words for marking the beginning of an interval, the plurality of special words including a first special word closest to the end of the sequence; Determining, from a target vocabulary in a vocabulary library, a word-gram combination that matches the target subsequence as a draft word-gram sequence for speculative decoding; wherein the vocabulary library includes a plurality of vocabulary tables corresponding to a plurality of intervals, and the target vocabulary table is a vocabulary table in the plurality of vocabulary tables corresponding to the interval represented by the first special word-gram; The draft word-gram sequence is verified using the large language model to determine a subsequent generated word-gram sequence.

2. The method according to claim 1, wherein The step of obtaining a target subsequence to be matched from the current word sequence includes: Sequentially concatenate the input word-gram sequence and the generated word-gram sequence generated by the large language model to obtain a prefix word-gram sequence; A first predetermined number of word-grams that do not belong to special word-grams are intercepted from the prefix word-gram sequence in order from back to front as target subsequences to be matched.

3. The method according to claim 1, wherein The step of determining, from a target vocabulary in a vocabulary library, a word combination matching the target subsequence as a draft word sequence for speculative decoding includes: Searching for a target term containing the target subsequence from each term included in the target vocabulary in descending order of frequency of occurrence; A word-gram combination of a second predetermined number of word-grams is intercepted from the end position of the target subsequence in the target word entry and is used as a draft word-gram sequence.

4. The method according to claim 1, wherein The using the large language model to verify the draft word sequence to determine a subsequent generated word sequence includes: Generate multiple candidate sequences based on the current word sequence and the draft word sequence; wherein a single candidate sequence includes the current word sequence, or includes a subsequence of the current word sequence and the draft word sequence taken from the beginning; Each candidate sequence is input into the large language model, and the large language model outputs a subsequent predicted word element for each candidate sequence; By comparing the predicted word with the word at the corresponding position of the draft word sequence, it is verified whether the word at the corresponding position of the draft word sequence is correct.

5. The method according to claim 4, wherein: The method further comprises: retaining each word before the first word verified as erroneous in the draft word sequence, and adding the retained word and the predicted word to the generated word sequence to obtain an updated generated word sequence; If the updated generated word-gram sequence does not include the second special word-gram representing the end of the interval, the iterative execution determines the target subsequence to be matched from the current word-gram sequence; if the updated generated word-gram sequence includes the second special word-gram, the large language model is triggered to predict the next word-gram.

6. The method according to claim 5, wherein: The method further comprises: If the updated generated word-gram sequence includes the second special word-gram, the word entries consisting of the word-grams in the interval are added to the target word list.

7. The method according to claim 6, wherein: The method further comprises: In response to the number of entries or the storage amount in the target vocabulary reaching a preset threshold, several entries with the lowest usage frequency are deleted.

8. The method of claim 1, wherein: The multiple vocabularies are pre-generated in the following manner: Initializing N cache spaces for storing the multiple word lists; For the knowledge text segments in the local knowledge base, the vocabulary corresponding to the interval is added according to the corresponding interval mark.

9. The method of claim 8, wherein: The knowledge text segments in the local knowledge base are generated by the large language model, and contain the special word-grams as the interval markers.

10. A device for generating text using a large language model, comprising: an acquisition unit, configured to acquire a target subsequence to be matched from a current word-gram sequence, wherein the current word-gram sequence includes an input word-gram sequence and a generated word-gram sequence generated by a large language model; the generated word-gram sequence includes a plurality of special words for marking the beginning of an interval, wherein the plurality of special words includes a first special word closest to the end of the sequence; a matching unit, configured to determine, from a target vocabulary in a vocabulary library, a word-gram combination that matches the target subsequence obtained by the obtaining unit as a draft word-gram sequence for speculative decoding; wherein the vocabulary library includes a plurality of vocabulary tables corresponding to a plurality of intervals, and the target vocabulary table is a vocabulary table in the plurality of vocabulary tables corresponding to the interval represented by the first special word-gram; A verification unit is used to verify the draft word-gram sequence obtained by the matching unit using the large language model to determine a subsequent generated word-gram sequence.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 9.

12. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 9 is implemented.

Images