Prospective sampling method and device for large language model and computer equipment

Abstract

The invention provides a speculation sampling method for a large language model, and the method comprises the steps: determining a target topic related to a target cue word from a plurality of preset topics after obtaining the target cue word of the input large language model; each target topic is provided with a corresponding word list, and after the target topic is determined, speculation sampling based on the frequency of each text fragment recorded in the word list can be carried out based on the target word list corresponding to the target topic. Therefore, prediction of the decoding stage of the large language model can be carried out by using a speculation sampling result. The speculation sampling based on the word list can reduce the occupation of video memory resources. And word lists of different topics are distinguished, so that the speculation sampling accuracy is improved.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer application technology, and more particularly, to a speculative sampling method, apparatus, and computer device for a large language model. Background Art

[0002] Large Language Model (LLM) is a deep learning model trained on massive text data. It can understand the meaning of text and generate natural language text based on given text, and handle various natural language tasks such as text summarization, question answering, translation, etc.

[0003] The process of inferring the output text by a large language model is slow. In order to solve this problem, the prior art generally adopts a speculative sampling (Speculative Decoding) algorithm, but the existing speculative sampling algorithm has the problem of occupying a large amount of video memory space and low accuracy. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a method, apparatus, and computer device for speculative sampling of a large language model.

[0005] According to a first aspect of one or more embodiments of this specification, a speculative sampling method for a large language model is proposed, comprising:

[0006] Obtain a target prompt word for input into the large language model;

[0007] Determining a target topic corresponding to the target prompt word from a plurality of preset topics; wherein any topic among the plurality of topics is provided with a corresponding vocabulary, which includes text segments generated by the large language model for the prompt word of the topic and the frequency of occurrence of each text segment;

[0008] According to the target vocabulary corresponding to the target topic, frequency-based speculative sampling is performed; the speculative sampling result is used for prediction of the large language model in the decoding stage.

[0009] According to a second aspect of one or more embodiments of this specification, a speculative sampling device for a large language model is proposed, comprising:

[0010] A target prompt word acquisition module, used to acquire the target prompt word input into the large language model;

[0011] A target topic determination module, used to determine the target topic corresponding to the target prompt word from a plurality of preset topics; wherein any topic among the plurality of topics is provided with a corresponding vocabulary, which includes text segments generated by the large language model for the prompt word of the topic and the frequency of occurrence of each text segment;

[0012] The speculative sampling module is used to perform frequency-based speculative sampling according to the target vocabulary corresponding to the target topic; the speculative sampling results are used for prediction of the large language model in the decoding stage.

[0013] According to a third aspect of an embodiment of this specification, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the speculative sampling method for a large language model as described in the first aspect of the embodiment of this specification is implemented.

[0014] According to a fourth aspect of an embodiment of this specification, a computer device is provided, the computer device comprising:

[0015] processor;

[0016] a memory for storing processor-executable instructions;

[0017] The processor implements the speculative sampling method for a large language model as described in the first aspect of the embodiment of this specification by running the executable instructions.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program product is provided. When the computer program product is executed by a processor, the speculative sampling method for a large language model described in the first aspect of the embodiments of this specification is implemented.

[0019] This specification provides a speculative sampling method for a large language model. After obtaining a target prompt word input into the large language model, a target topic related to the target prompt word is determined from multiple preset topics. Each target topic is provided with a corresponding vocabulary. After the target topic is determined, speculative sampling based on the frequency of each text fragment recorded in the vocabulary can be performed based on the target vocabulary corresponding to the target topic. Thus, the speculative sampling results can be used to predict the decoding stage of the large language model. Speculative sampling based on the vocabulary can reduce the occupation of video memory resources. The vocabulary of different topics is distinguished to improve the accuracy of speculative sampling.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0022] Figure 1 It is a schematic diagram of speculative sampling shown in this specification.

[0023] Figure 2It is a flowchart of a speculative sampling method for a large language model according to an exemplary embodiment of this specification.

[0024] Figure 3 This is a flowchart of a speculative sampling method for a large language model according to a specific embodiment of this specification.

[0025] Figure 4 It is a block diagram of a speculative sampling device for a large language model according to an exemplary embodiment of the present specification.

[0026] Figure 5 It is a hardware structure diagram of a computer device shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0028] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0029] By using LLM, users can input prompts and get the corresponding output text. During the reasoning process of LLM, the prompt is first converted into a sequence of text units (tokens), and the text unit sequence is encoded into a vector. Then, prediction is performed based on the vector corresponding to the prompt (hereinafter referred to as decoding or reasoning). During the prediction process, the output text is generated through multiple cycles, and a token is generated in each cycle. In each cycle, the next token of the output text is generated based on the token of the output text that has been generated and the token corresponding to the prompt. Among them, token refers to the basic unit of model processing, which usually corresponds to one or more characters.

[0030] In the above reasoning process, the generation of each token depends on the generated tokens, which means that the prediction process must be executed serially, which in turn makes the LLM reasoning process slow and users have to wait for a long time.

[0031] In order to solve the problem of slow reasoning process, methods such as speculative sampling are often used to speed up the reasoning process of LLM. Among them, speculative sampling means: before performing reliable first-type calculations, a set of results are guessed through the second-type calculation method, and then the correctness of the guessed results is verified by the first-type calculation process. The first-type calculation can be a prediction process executed by LLM that consumes more computing resources and has a higher confidence level, while the second-type calculation is a simpler calculation process with relatively lower confidence.

[0032] In a large language model, a speculative sampling algorithm can be added in each cycle to accelerate the LLM reasoning process. Specifically, in each cycle, before using LLM to predict the next token, the speculative sampling method can be used to predict the subsequent multiple tokens of the currently generated token, and the accuracy of the generated multiple tokens can be verified by LLM.

[0033] For example, Figure 1 As shown in the figure, each solid line block represents a token predicted by LLM. The five tokens following T3, namely T4-T8, can be predicted by speculative sampling. Then, T1-T3, T1-T4, T1-T5, ... T1-T8 can be input into LLM respectively, and LLM can perform inference on them concurrently in one cycle. LLM can generate a subsequent token (T4) by reasoning about T1-T3. Since the subsequent token of T1-T3 generated by LLM is the same as the subsequent token of T1-T3 obtained by speculative sampling, it can be determined that T4 obtained by speculative sampling is accurate. LLM can generate a subsequent token (T5) by reasoning about T1-T4. Similarly, it can be confirmed that T5 obtained by speculative sampling is accurate. By analogy, it can be determined that the speculative sampling results of T4-T6 are accurate, while the speculative sampling results of T7 and T8 are inaccurate, and then confirm the output of T1-T6, and discard T7 and T8. Compared with the original LLM which can only generate one T4 in one cycle, speculative sampling can generate additional T5 and T6 in one cycle, thus improving efficiency.

[0034] Compared with the general prediction process of LLM, which can only generate one token in one cycle, speculative sampling allows multiple tokens to be generated in one cycle. When the accuracy of multiple tokens is high, the number of LLM cycles can be reduced. In addition, the computational cost of speculative sampling is generally much lower than the computational cost of running LLM once, so additional speculative sampling can speed up the reasoning process of LLM. It should be noted that although the use of speculative sampling will cause LLM to process more content in one cycle, due to the high concurrency of the processor, LLM processes more content in one cycle with almost no delay, thus ensuring generation efficiency.

[0035] Next, we will briefly describe an optional speculative sampling method. This method is a speculative sampling algorithm that uses a small model as a draft model. Specifically, a small model is distilled from the LLM and used as a speculative sampling method to predict tokens.

[0036] This method has the following shortcomings:

[0037] First, the prediction of the small model requires more video memory resources. For example, for a 5-byte small model distilled from a 72-byte large model, the coefficients of the small model still occupy 10 GB of video memory. Moreover, in scenarios with long contexts, several GB of video memory is required to store the attention information of the context.

[0038] Second, when you need to increase the number of tokens predicted by speculative sampling, it will cause a large delay. Since the small model is a model distilled from the large model, it has the same properties as the large model, that is, only one token can be generated in one cycle. If you want to increase the number of tokens predicted by speculative sampling, you will need to use the small model for multiple inferences, which still takes a lot of time. For example, if you need to predict 5 tokens through speculative sampling, even if one cycle only takes 5-10ms, it will still cause a delay of 25-50ms.

[0039] In order to solve the above problems, this specification provides a speculative sampling method for a large language model. After obtaining the target prompt word input into the large language model, a target topic related to the target prompt word is determined from multiple preset topics. Each target topic is provided with a corresponding vocabulary. After the target topic is determined, speculative sampling based on the frequency of each text segment recorded in the vocabulary can be performed based on the target vocabulary corresponding to the target topic. Thus, the speculative sampling results can be used to predict the decoding stage of the large language model.

[0040] The method provided in this specification found that the speculative sampling process is a process that is strongly related to the topic, and the frequency statistics of text fragments of different topics will interfere with each other, which will lead to a low accuracy of speculative sampling. For example, prompt is related to football, but the content obtained by speculative sampling includes professional vocabulary related to basketball. Therefore, this specification distinguishes the word lists of different topics to avoid mutual interference between the frequency statistics of different topics, thereby improving the accuracy of speculative sampling.

[0041] Moreover, if this method needs to increase the length of speculative sampling, it only needs to increase the length of the text fragment recorded in the vocabulary, without any additional time cost.

[0042] In addition, since the speculative sampling process based on the vocabulary has a low degree of concurrency, and the GPU is more suitable for processing high-concurrency tasks, the speculative sampling process does not need to be completed inside the GPU. This means that speculative sampling does not need to occupy a lot of video memory resources.

[0043] Next, a speculative sampling method for a large language model provided in this specification will be described in detail.

[0044] Before executing the method, the vocabulary corresponding to each of the multiple topics may be obtained. Next, a method for determining multiple topics and a method for obtaining the vocabulary corresponding to the multiple topics will be described through a specific embodiment. This embodiment does not limit the present specification.

[0045] In an optional implementation, multiple topics may be preset with corresponding representation vectors. The method for obtaining the representation vectors corresponding to the preset multiple topics may be to cluster the representation vectors corresponding to a large number of prompt words to obtain multiple clusters, and use the cluster centers of each cluster as the representation vectors corresponding to each topic.

[0046] In other words, the representation vectors corresponding to the multiple prompt words can be obtained; the representation vectors corresponding to the multiple prompt words are clustered to obtain multiple cluster centers, and the vectors corresponding to the multiple cluster centers are used as the representation vectors corresponding to the multiple topics.

[0047] This stage can be performed at the service initialization stage. Specifically, when the service is just being used, speculative sampling can be skipped and prompt words input by users can be collected. The collected prompt words of users are clustered to obtain N clusters. The vectors of the cluster centers corresponding to the N clusters are used as the representation vectors corresponding to the multiple topics.

[0048] In an optional implementation, during the clustering process, the distance / similarity between the characterization vectors may be calculated based on the inner product.

[0049] The number N of clusters can be preset, and can be specifically determined based on the requirements of computational complexity and accuracy. As the number of clusters increases, it is necessary to calculate the similarity between the target representation vector and the representation vectors corresponding to more topics when matching in step 203 later. At the same time, the increase in the number of clusters will also lead to more refined topic classification, making the accuracy of speculative sampling during reasoning higher. Therefore, the number N of clusters can be comprehensively determined based on the requirements of LLM for computational complexity and accuracy.

[0050] The representation vectors corresponding to the multiple prompt words are obtained in the same way as the target representation vector described later, and both can be extracted based on the neural network model that encodes the text. The representation vectors corresponding to the multiple prompt words can also be obtained by: obtaining the representation vectors output for the multiple prompt words in the prefill phase of the large language model. In this way, while the large language model is executing the reasoning process for the prompt words, the representation vectors corresponding to the multiple prompt words can be generated using the large language model, saving the amount of calculation.

[0051] In addition, as mentioned above, each of the multiple topics is provided with a corresponding vocabulary. Next, the generation of the target vocabulary and the specific composition of the target vocabulary will be described by taking the target vocabulary corresponding to the target topic as an example.

[0052] As mentioned above, the target word list may include text segments generated by prompt words belonging to the target topic, and the frequency of occurrence of each text segment in the output text corresponding to the target topic. The text segment may be of a preset length, such as M tokens.

[0053] The method for obtaining the vocabulary corresponding to any topic can be: obtaining the text generated by the large language model for the prompt words of the arbitrary topic; from the generated text, cutting multiple text fragments of length M, storing them in the vocabulary corresponding to the topic, and recording the frequency of occurrence of the corresponding text fragments in the vocabulary.

[0054] In other words, taking the target vocabulary as an example, the method for obtaining the target vocabulary can be: obtaining multiple output texts output by LLM for each prompt word corresponding to the target topic, intercepting M consecutive tokens from each output text as a text segment, and obtaining multiple text segments corresponding to the output text by intercepting. After obtaining the text segments, the number of occurrences / frequency of each text segment can be counted, and the frequency data and the text segment can be recorded in the vocabulary according to the corresponding relationship.

[0055] Next, we will use a specific example to illustrate the method of extracting text fragments. For example, for a prompt word, LLM outputs the following content: "Today is sunny, it is recommended to go to the park." Assuming that one word corresponds to one token, and M is 10, then the following results can be obtained through extraction: (1) "Today is sunny, it is recommended to go to the park"; (2) "It is sunny, it is recommended to go to the park"; (3) "It is sunny, it is recommended to go to the park". The above results can be stored as three text fragments in the vocabulary.

[0056] Next, we will combine Figure 2 The speculative sampling method for a large language model shown in this specification is described below. Figure 2 As shown, the following steps are included:

[0057] Step 201, obtaining a target prompt word input into the large language model.

[0058] The prompt word is the question text input by the user to the large model. The target prompt word can be any unprocessed prompt word. The method of this specification is explained through the processing process of the target prompt word.

[0059] After obtaining the target prompt word, the representation vector output by the Prefill stage of the large language model for the target prompt word can also be obtained. The LLM reasoning process includes two stages, namely the Prefill stage and the decoding stage. In the Prefill stage, LLM can convert the input prompt word into the form of a representation vector and generate the first token of the output text based on the prompt word. In the decoding stage, LLM can generate the output text through multiple cycles.

[0060] Step 203: determining a target topic corresponding to the target prompt word from a plurality of preset topics.

[0061] Among them, any topic among the multiple topics is provided with a corresponding vocabulary, which includes text segments generated by the large language model for the prompt words of the topic and the frequency of occurrence of each text segment.

[0062] Specifically, in order to ensure the accuracy of speculative sampling, topics related to the target cue word or to which the target cue word belongs can be determined.

[0063] The method for determining the target topic will be described here by means of an embodiment, which does not limit the present specification.

[0064] Specifically, a target representation vector corresponding to the target prompt word may be extracted; the similarity between the target representation vector and the representation vectors corresponding to each of the multiple topics may be calculated, and the target topic may be determined based on the similarity calculation result.

[0065] Among them, in order to reduce the computational complexity, the target representation vector can be extracted through the pre-fill stage of the LLM as described above. Optionally, the target representation vector can also be extracted by other neural network models that can encode text. The target topic can be the topic with the highest similarity to the target representation vector, or any topic with a similarity greater than a preset threshold to the target representation vector. For example, it can be the topic with the highest sum of the occurrence frequencies of the text segments in the corresponding vocabulary that has a similarity greater than the preset threshold to the target representation vector.

[0066] By converting the target prompt and multiple topics into the form of representation vectors and then determining the target topic according to the similarity between the representation vectors, the target topic can be determined more conveniently and quickly.

[0067] Step 205: Perform frequency-based speculative sampling according to the target vocabulary corresponding to the target topic.

[0068] Among them, the speculative sampling result is used for the prediction of the large language model in the decoding stage.

[0069] Specifically, after obtaining the target vocabulary, several tokens already generated by the LLM can be matched with the vocabulary to obtain multiple matched text segments. One text segment with the highest frequency can be selected from the multiple matched text segments as the speculative sampling result.

[0070] One or more text segments with a frequency greater than a preset threshold can also be selected from the multiple matched text segments as the matching result. As described above, due to the concurrent processing characteristics of the LLM, inputting multiple text segments of speculative sampling at one time for accuracy discrimination will not affect the latency of one cycle of the LLM when the number of text segments is appropriate. Therefore, this method will not affect the execution efficiency of the LLM.

[0071] After obtaining the speculative sampling result, the LLM can be used to determine whether the speculative sampling result is accurate, thereby determining the generated token. As described above, in the decoding stage of the LLM, the process of speculative sampling - LLM model inference is repeated multiple times to complete the generation of the output text. Therefore, the above step 205 can be executed in a loop until the output text is obtained.

[0072] Among them, in an optional implementation manner, the frequency-based speculative sampling can be: assuming that the length of each text segment in the vocabulary is M, then the last one, the last two... the last M - 1 tokens in the tokens already generated by the LLM can be respectively input into the vocabulary for matching to obtain multiple matched text segments. Then, the speculative sampling result can be selected from them.

[0073] Among them, vocabulary matching is not a high-concurrency task, and GPU is generally more suitable for processing high-concurrency tasks, and video memory space should be reserved for high-concurrency tasks. Therefore, the vocabulary can be stored in the memory to occupy as little precious video memory resources as possible.

[0074] In addition, after the large language model completes the inference for the target prompt word, the predicted text generated for the target prompt word in the decoding stage of the large language model can also be obtained, and the target word list can be updated based on the predicted text. In other words, the target word list can be updated by the text generation result corresponding to the target prompt word.

[0075] Next, the speculative sampling method for a large language model shown in this specification will be described in conjunction with a specific embodiment. The method may include two stages, namely an offline preprocessing stage and an online service stage.

[0076] 1. Offline preprocessing stage.

[0077] First, collect prompt examples of large language models from online applications, which can collect tens of thousands to hundreds of thousands of prompts.

[0078] Second, the collected prompts are input into LLM for the prefill phase, and the hidden states of the last layer are obtained to obtain the representation vector of each prompt.

[0079] Third, cluster the representation vectors of all prompts to obtain N clusters.

[0080] Fourth, the cluster centers of the N clusters are used to form a matrix of size N*hidden_size, where hidden_size is the length of prompt. This matrix is ​​recorded as the topic classification matrix C. In other words, the matrix C includes representation vectors corresponding to multiple topics.

[0081] 2. Online service stage.

[0082] For specific operations performed during the online service phase, see Figure 3 .like Figure 3 As shown, this stage includes two sub-stages, namely the topic classification stage and the topic classification vocabulary prediction and updating stage.

[0083] In the topic classification stage, the prompt input by the user is the target prompt mentioned above. After the target prompt is processed in the pre-filling stage of LLM, the target representation vector can be obtained. The target representation vector and the topic classification matrix C are subjected to matrix multiplication (MatMul) to obtain the similarity between the target representation vector and the representation vectors corresponding to each topic (expressed by the inner product). Furthermore, the target topic that is most relevant to the target prompt word among multiple topics can be determined through the maximum independent variable point set (ArgMax), which is recorded as topic k.

[0084] The matrix C can be stored in the video memory space, which is convenient for using the GPU for matrix multiplication operations. The topic classification matrix generally only occupies a few hundred KB of video memory resources. Compared with the speculative sampling method using a small model or MLP mentioned above, it still achieves the effect of reducing the occupation of video memory resources.

[0085] In the stage of predicting and updating the topic classification vocabulary, in any round of the cycle, speculative sampling is performed from the vocabulary of topic k to obtain m tokens. The decoding process is performed based on the speculative sampling results. The results of the decoding process are used to update the vocabulary of topic k until the decoding process is completed.

[0086] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of a device and a terminal to which it is applied.

[0087] like Figure 4 As shown, Figure 4 : is a block diagram of a speculative sampling device for a large language model according to an exemplary embodiment of the present specification, the device comprising:

[0088] A target prompt word acquisition module 410, used to acquire a target prompt word input into the large language model;

[0089] A target topic determination module 420 is used to determine a target topic corresponding to the target prompt word from a plurality of preset topics; wherein any topic among the plurality of topics is provided with a corresponding vocabulary, which includes text segments generated by the large language model for the prompt word of the topic and the frequency of occurrence of each text segment;

[0090] The speculative sampling module 430 is used to perform frequency-based speculative sampling according to the target vocabulary corresponding to the target topic; the speculative sampling result is used for the prediction of the large language model in the decoding stage.

[0091] In an optional implementation, the target topic determination module 420 includes:

[0092] A target representation vector extraction submodule 421 (not shown in the figure) is used to extract the target representation vector corresponding to the target prompt word;

[0093] The target topic determination module 422 (not shown in the figure) is used to calculate the similarity between the target representation vector and the representation vectors corresponding to each of the multiple topics, and determine the target topic according to the similarity calculation result.

[0094] In an optional implementation, the device further includes a characterization vector determination module 400 (not shown in the figure), including:

[0095] The representation vector acquisition submodule 401 (not shown in the figure) is used to obtain the representation vectors corresponding to the plurality of prompt words respectively;

[0096] The clustering submodule 402 (not shown in the figure) is used to cluster the representation vectors corresponding to the multiple prompt words to obtain multiple cluster centers, and use the vectors corresponding to the multiple cluster centers as the representation vectors corresponding to the multiple topics.

[0097] In an optional implementation, the representation vector acquisition submodule 401 (not shown in the figure) is specifically used to obtain the representation vectors output for multiple prompt words in the prefill stage of the large language model.

[0098] In an optional implementation, the target representation vector extraction submodule 421 (not shown in the figure) is specifically used to obtain the representation vector output by the Prefill stage of the large language model for the target prompt word.

[0099] In an optional implementation, a vocabulary updating module 440 (not shown in the figure) is further included, which is used to obtain the predicted text generated for the target prompt word in the decoding stage of the large language model, and update the target vocabulary based on the predicted text.

[0100] In an optional embodiment, the device also includes a vocabulary building module 440 (not shown in the figure), which is used to obtain the text generated by the large language model for the prompt words of the arbitrary topic; from the generated text, multiple text fragments of length M are intercepted, stored in the vocabulary corresponding to the topic, and the frequency of occurrence of the corresponding text fragments is recorded in the vocabulary.

[0101] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0102] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0103] like Figure 5 As shown, Figure 5 A hardware structure diagram of a computer device in which a speculative sampling device for a large language model is located in an embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other in communication within the device through the bus 1050.

[0104] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above method by running executable instructions.

[0105] The memory 1020 for storing processor executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020.

[0106] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0107] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0108] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0109] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0110] The embodiments of this specification also provide a computer program product, which, when executed by a processor, implements the speculative sampling method for a large language model described in the first aspect of the embodiments of this specification.

[0111] The embodiments of this specification also provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the speculative sampling method for a large language model described in the first aspect of the embodiments of this specification is implemented.

[0112] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0114] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0115] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A speculative sampling method for a large language model, comprising: Obtain a target prompt word for input into the large language model; Determining a target topic corresponding to the target prompt word from a plurality of preset topics; wherein any topic among the plurality of topics is provided with a corresponding vocabulary, which includes text segments generated by the large language model for the prompt word of the topic and the frequency of occurrence of each text segment; Performing frequency-based speculative sampling according to a target vocabulary corresponding to the target topic; The speculative sampling results are used for prediction of the large language model in the decoding stage.

2. The method according to claim 1, wherein: The plurality of topics are each preset with a corresponding representation vector, and determining a target topic corresponding to the target prompt word from the preset plurality of topics includes: Extract the target representation vector corresponding to the target prompt word; The similarity between the target representation vector and the representation vectors corresponding to the multiple topics is calculated, and the target topic is determined according to the similarity calculation result.

3. The method according to claim 2, wherein: The representation vectors corresponding to the multiple topics are determined in the following manner: Obtain the representation vectors corresponding to multiple prompt words respectively; The representation vectors corresponding to the multiple prompt words are clustered to obtain multiple cluster centers, and the vectors corresponding to the multiple cluster centers are used as the representation vectors corresponding to the multiple topics.

4. The method according to claim 3, wherein: The step of obtaining the representation vectors corresponding to the plurality of prompt words includes: Obtain representation vectors output for multiple prompt words in a prefill phase of the large language model.

5. The method according to claim 2, wherein: The extracting the representation vector corresponding to the target prompt word includes: Obtain a representation vector output by the Prefill stage of the large language model for the target prompt word.

6. The method according to claim 1, wherein: The method further comprises: The predicted text generated for the target prompt word in the decoding stage of the large language model is obtained, and the target word list is updated based on the predicted text.

7. The method according to claim 1, wherein: The vocabulary corresponding to any topic is constructed in the following way: Get the text generated by the large language model for the prompt word of the arbitrary topic; From the generated text, multiple text segments of length M are intercepted and stored in the vocabulary corresponding to the topic, and the frequency of occurrence of the corresponding text segments is recorded in the vocabulary.

8. A speculative sampling device for a large language model, comprising: A target prompt word acquisition module, used to acquire the target prompt word input into the large language model; A target topic determination module, used to determine the target topic corresponding to the target prompt word from a plurality of preset topics; wherein any topic among the plurality of topics is provided with a corresponding vocabulary, which includes text segments generated by the large language model for the prompt word of the topic and the frequency of occurrence of each text segment; The speculative sampling module is used to perform frequency-based speculative sampling according to the target vocabulary corresponding to the target topic; the speculative sampling results are used for prediction of the large language model in the decoding stage.

9. A computer program product, wherein when the computer program product is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 7 by running the executable instructions.

Images