Inference method, processing method and electronic equipment
By using a subset of the vocabulary and a dynamic update mechanism in the encoding and decoding stages of the large language model, the problem of excessive computing resources on the edge device is solved, the inference efficiency and accuracy are improved, and the resource requirements are reduced.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- LENOVO (BEIJING) LTD
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-10
AI Technical Summary
Large language models deployed on the edge face the problem of ultra-large matrix operations during the inference stage, resulting in an excessive burden on disk storage, memory relocation, and computing resources.
Encoded tokens are generated using a first vocabulary, and a subset of the second vocabulary is used for decoding output during the decoding stage. The second vocabulary is dynamically updated to include missing tokens. Intermediate tokens are generated during speculative decoding by combining the third vocabulary. The vocabulary is optimized to reduce computational load and resource requirements.
It reduces the computational load and resource requirements of edge devices, improves inference efficiency and accuracy, and reduces the requirements for disk storage and memory relocation.
Smart Images
Figure CN121835903A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of large language models, and particularly relates to a reasoning method, a processing method and an electronic device. BACKGROUND
[0002] At present, large language models (LLM) deployed on the edge side indeed face the problem of super-matrix operation in the reasoning stage, which causes a great burden on the disk storage, memory movement and computing resources of the edge device. SUMMARY
[0003] In order to solve the above technical problems, the purpose of the present application is to provide a reasoning method, a processing method and an electronic device.
[0004] The technical solutions adopted are as follows: A reasoning method comprises: obtaining a user input prompt word, in the encoding stage of a language model, processing the prompt word based on a first vocabulary to generate an encoded token; in the decoding stage of the language model, obtaining a decoded output token based on a second vocabulary; wherein the second vocabulary is a subset of the first vocabulary.
[0005] In some embodiments, the reasoning method further comprises: if a target token in the encoded token does not exist in the second vocabulary, adding the target token to the second vocabulary to update the second vocabulary.
[0006] In some embodiments, the reasoning method further comprises: obtaining an intermediate token generated in the decoding stage reasoning process, if the intermediate token does not exist in the second vocabulary, adding the intermediate token to the second vocabulary to update the second vocabulary.
[0007] In some embodiments, the reasoning method further comprises: in the multiple-session reasoning process of the language model, obtaining usage information of each token in the updated second vocabulary; determining a ranking position based on the usage information, and selecting tokens of a preset ranking threshold length to form a target vocabulary; wherein the number of tokens in the target vocabulary is less than the number of tokens in the second vocabulary.
[0008] In some embodiments, the decoding stage comprises a speculative decoding process, and the reasoning method further comprises: in the speculative decoding process of the language model, generating the intermediate token based on a third vocabulary, The third vocabulary is a subset of the second vocabulary or identical to the second vocabulary.
[0009] In some embodiments, the intermediate token is a guessed token in a speculative decoding process, If the guessed token does not exist in the second vocabulary, both of the two tokens in the token pair formed by the guessed token and the predicted token corresponding to the guessed token are added to the second vocabulary to update the second vocabulary.
[0010] In some embodiments, the decoding stage includes a speculative decoding process, and the inference method further includes: In a language model speculative decoding process, a first prediction result is generated based on a third vocabulary, the prediction result including a candidate sequence of multiple consecutive tokens; the third vocabulary is a subset of the second vocabulary or identical to the second vocabulary; The first prediction result is verified based on the first vocabulary to obtain a second prediction result corresponding to the first prediction result, and the token at each position in the first prediction result is compared with the corresponding token in the second prediction result; When the first token at the first position in the first prediction result does not match the second token at the corresponding position in the second prediction result, the second token at the corresponding position in the second prediction result is added to the second vocabulary as the intermediate token.
[0011] In some embodiments, the initial state of the second vocabulary is a pruned vocabulary based on the first vocabulary, wherein at least one subset is selected from the first vocabulary to construct the second vocabulary based on output statistical information generated when the language model is processed offline on a general text data set.
[0012] A processing method, comprising: obtaining a target token generated by a language model in an encoding process and / or a decoding process in response to user input; obtaining a second vocabulary used for decoding an output layer vector to obtain an output token, wherein the second vocabulary has an initial pruning size; If the target token in the encoding token does not exist in the second vocabulary, adding the target token to the second vocabulary to update the second vocabulary, and / or; obtaining an intermediate token generated by the decoding stage inference process, If the intermediate token does not exist in the second vocabulary, adding the intermediate token to the second vocabulary to update the second vocabulary.
[0013] An electronic device comprises: a memory configured to store a language model; a processor configured to obtain a user input prompt word, in an encoding stage of the language model, process the prompt word based on a first vocabulary, and generate an encoded token; in a decoding output stage of the language model, obtain a decoded output token based on a second vocabulary, wherein the second vocabulary is a subset of the first vocabulary. BRIEF DESCRIPTION OF DRAWINGS
[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, and the advantages thereof, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on these drawings.
[0015] Figure 1 A flowchart of a reasoning method provided by Embodiment One of the present application; Figure 2 A flowchart of a reasoning method provided by Embodiment Two of the present application; Figure 3 A flowchart of a reasoning method provided by Embodiment Three of the present application; Figure 4 A flowchart of a reasoning method provided by Embodiment Four of the present application; Figure 5 A flowchart of a processing method provided by Embodiment One of the present application; Figure 6 A flowchart of a processing method provided by Embodiment Two of the present application; Figure 7 A flowchart of a processing method provided by Embodiment Three of the present application; Figure 8 A flowchart of a processing method provided by Embodiment Four of the present application; Figure 9 A flowchart of a processing method provided by Embodiment Five of the present application; Figure 10 A flowchart of a processing method provided by Embodiment Six of the present application. DETAILED DESCRIPTION
[0016] In order to further illustrate the technical means and effects taken by the present application to achieve the predetermined purpose, the specific implementation, structure, features and effects of the reasoning method, processing method and electronic device according to the present application are described in detail below in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.
[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
[0018] Artificial intelligence deeply empowers different scenarios and is increasingly widely used in learning, work and life. Currently, on Android devices, users are increasingly expecting end-side devices with closer interactions of technologies such as artificial intelligence to provide more convenient and intelligent operations and experiences for people.
[0019] However, since the current excellent large language model (LLM) has a parameter magnitude of hundreds of millions or even tens of billions, it consumes a lot of disk space and on-chip chip computing power. In order to reduce the dependence on disk space and chip computing power and successfully land on resource-limited end-side devices, it is necessary to reduce the amount of calculation and the computing power requirement of the end-side devices.
[0020] It should be noted that the model involved in the present application can be a general AI (Artificial Intelligence) model, which can use but is not limited to Transformer or its architecture variants (such as using only encoder (Encoder-Only) / decoder (Decoder-Only), encoder-decoder (Encoder-Decoder), or MoE (Mixture of Experts, a neural network architecture) and other basic architectures) or other basic architectures, to learn the characteristics and rules of natural language by training a large amount of diversified data, so as to be able to understand and generate natural language. It usually has hundreds of millions to tens of billions of model parameters (model parameters are variables that control the behavior of the target model), which can capture the complex relationships and patterns in natural language.
[0021] To solve the above technical problems, the specific scheme of the reasoning method provided by the present application is described in detail below in combination with the drawings.
[0022] As shown in Figure 1 The reasoning method provided by the present application comprises: Step S100: obtaining a user input prompt word; One method for obtaining input prompts is through a text input box. For example, a text box can be provided on the terminal device, where users can enter or paste prompts.
[0023] Alternatively, the user's voice can be converted into prompts using the voice recognition technology provided by the terminal device. For example, when the voice assistant on the terminal device speaks a command, the terminal device will convert the received command into the corresponding prompt.
[0024] Step S200: In the encoding stage of the language model, the prompt words are processed based on the first vocabulary to generate encoded lexical units; The first vocabulary is used to convert prompt words into digital sequences that the language model can process; these digital sequences are the encoded lexical units.
[0025] The first vocabulary includes at least a first set of lexical units, segmentation rules, and lexical unit mapping relationships. The first set of lexical units can be a list containing multiple lexical units, each of which can be a word, subword, character, or symbol. The size of the lexical unit set (vocabulary size) is usually predefined, for example, 128,256 lexical units.
[0026] Token segmentation rules are used to divide input text or prompt words into sequences of tokens. Token segmentation rules can be at least one of byte pair encoding rules, space-based segmentation rules, and WordPiece rules. The token mapping relationship assigns a unique positive integer identifier, or Token ID, to each token.
[0027] In step S200, after obtaining the prompt word, the first vocabulary can be loaded in the encoding stage, and the input prompt word can be segmented into a word element sequence using the word segmentation rules of the first vocabulary. Each word element in the word element sequence is assigned a Token ID, thereby generating an encoded word element sequence.
[0028] For example, when the input prompt is "peach blossoms bloom in spring," the input word can be segmented into three word units: ["peach blossoms," "spring equinox," "blooming"]. Then, each of these three word units is assigned an ID. For instance, "peach blossoms" corresponds to ID 1245, "spring equinox" to ID 5678, and "blooming" to ID 7012. The final output encoded word units would then be [1245, 5678, 7012], and this ID sequence is the encoded word unit.
[0029] Step S300: In the decoding stage of the language model, decoded output lexical units are obtained based on the second lexical list; wherein the second lexical list is a subset of the first lexical list.
[0030] The second vocabulary is used to convert the numerical vectors output by the language model into readable text tokens. Compared with the first vocabulary, the second vocabulary also has a token set and a mapping relationship, but it is different from the first vocabulary in that, in the decoding stage, the language model outputs a series of vectors, such as the vector sequence 【1245, 5678, 7012】 in the foregoing. These vector sequences carry semantic information. The role of the second vocabulary is to convert the vector sequence into a specific token, that is, to translate it into a peach blossom in spring.
[0031] It can be understood that the role of the first vocabulary is to convert a specific token into a vector sequence, and the role of the second vocabulary is to convert a vector sequence into a specific token.
[0032] In this embodiment, the second vocabulary can be a subset of the first vocabulary. It can be understood that the number of tokens in the second vocabulary is less than the number of tokens in the first vocabulary, and the tokens in the second vocabulary are all from the tokens in the first vocabulary. In the decoding process, appropriate tokens are selected from the second vocabulary as decoding output tokens. By using the second vocabulary as the basis for decoding output, the number of tokens that need to be processed in the decoding stage can be reduced, thereby reducing the amount of calculation and the demand for computing power of the end-side device.
[0033] In actual applications, the usage frequency or occurrence probability of the tokens in the first vocabulary can be counted, and tokens with higher usage frequency or occurrence probability can be selected to form the second vocabulary. This formation method of the second vocabulary can ensure that the second vocabulary contains tokens that are more likely to appear in actual applications, thereby improving the efficiency and accuracy of decoding.
[0034] Alternatively, clustering algorithms or the like can be used to cluster the tokens in the first vocabulary, and tokens with similar semantics can be classified into a class. Then, representative tokens are selected from each cluster to form the second vocabulary. In this way, the number of tokens can be reduced while retaining the semantic diversity of the tokens.
[0035] The inference method of the embodiment of the present application obtains decoding output tokens through the second vocabulary in the decoding stage of the model. Since the number of tokens in the second vocabulary is less than that in the first vocabulary, the process of calculating the probability of each token is faster, which reduces the response time of the model and in turn reduces the amount of calculation of the model in the decoding stage. In the decoding process, the model does not need to calculate and screen all the tokens in the first vocabulary, but only operates within the range of the second vocabulary, thereby effectively reducing the computational complexity and improving the inference efficiency.
[0036] In addition, the use of the second vocabulary can also reduce the requirements for disk storage, memory relocation, and computing power resources of the end-side device. As shown in FIG. 8, in some embodiments, the inference method can further include: Figure 2 Step 201: If the target token in the encoded token does not exist in the second vocabulary; Step 202: Add the target token to the second vocabulary to update the second vocabulary.
[0037] Wherein, the target token is any one or more token units in the encoded token sequence generated by processing the input prompt word based on the first vocabulary, which does not exist in the current second vocabulary. These target tokens are usually professional terms, emerging vocabulary, specific field concepts or user-defined expressions.
[0038] In step 201, at system initialization, a mapping table from the ID of the first vocabulary to the ID of the second vocabulary can be constructed, and for each token in the first vocabulary, find its ID in the second vocabulary. If it can be found, record the mapping; if not, mark it as missing, that is, the token that does not find the corresponding ID in the second vocabulary is the target token, and in the subsequent reasoning process, when the new input prompt word is encoded to generate the encoded token sequence, the mapping table can be used to quickly determine which encoded token is the target token.
[0039] For example, assume that the input prompt word obtained is "peach blossom spring division blooming", and the first vocabulary processes the prompt word to generate the encoded token 【1245, 5678, 7012】. Among them, the token "blooming" corresponds to the ID 7012, and it is found that there is no ID corresponding to this token in the second vocabulary when constructing the mapping table, at this time the token "blooming" can be marked as a target token.
[0040] In step 202: After determining the target token, it is added to the second vocabulary. Among them, the process of adding can be to append the token at the end of the second vocabulary and assign it a new ID. For example, if the second vocabulary originally has 7011 tokens, the ID range is 1 - 7011, then the newly added "blooming" token can be assigned ID 7012.
[0041] In this way, the updated second vocabulary contains this originally missing target token, and in the subsequent decoding stage, the model can process the token to avoid errors or incomplete output due to missing tokens.
[0042] At the same time, the process of updating the second vocabulary also updates the mapping table to record the new mapping relationship between the target token in the first vocabulary and the updated second vocabulary, so that it can be accurately found and used in subsequent reasoning.
[0043] As shown in Figure 3 In some embodiments, the reasoning method can further include: Step 301: Obtain the intermediate token generated in the decoding stage reasoning process; Step 302: if the intermediate word does not exist in the second vocabulary; Step 303: add the intermediate word to the second vocabulary to update the second vocabulary.
[0044] In step 301, the intermediate word can be obtained in various ways in the decoding stage, such as a plurality of candidate words generated quickly by a draft model in the speculative decoding stage, which can be an intermediate word; Or, in each step of decoding, the language model outputs a word probability distribution to represent the possibility of each word as the next output. In the sampling process, the top K words with higher probabilities in the model output probability distribution are obtained, which can also be intermediate words.
[0045] In step 302, it can be determined whether the intermediate word exists by traversing the second vocabulary. For example, the intermediate word is compared with each word in the second vocabulary, and if no matching word is found in the second vocabulary, it is determined that the intermediate word does not exist in the second vocabulary.
[0046] For example, assuming the intermediate word is "shengkai", and no such word is found in the traversal of the second vocabulary, it is determined that the word "shengkai" is an intermediate word that does not exist in the second vocabulary.
[0047] In step 303, when it is determined that the intermediate word does not exist in the second vocabulary, the intermediate word is added to the end of the second vocabulary like the target word, and a new ID is assigned to it. In the process of multiple session reasoning of the language model, as the second vocabulary is constantly updated, the number of words will gradually increase, which may affect the reasoning efficiency. Therefore, the updated second vocabulary also needs to be optimized.
[0048] As shown in Figure 4 In some embodiments, the reasoning method can further include: Step 400: obtaining the usage information of each word in the updated second vocabulary in the process of multiple session reasoning of the language model; In step 400, the usage information can be the usage frequency and usage time of each word. Among them, the usage frequency refers to the number of times a word is used in multiple session reasoning; the usage time refers to the specific time when the word is used.
[0049] Step 401: determine the sorting position based on the usage information, select the words with a preset sorting threshold length to form a target vocabulary; finally, the number of words in the stable target vocabulary is less than the number of words in the second vocabulary, and the reasoning is based on the target vocabulary, which further reduces the calculation amount of the model and can maintain a high model reasoning generalization degree.
[0050] In step 401, the manner of determining the sorting position based on the usage information can be to sort from high to low according to the usage frequency of the word units, and the word units with high usage frequency are arranged in front, and the word units with low usage frequency are arranged in back.
[0051] The preset sorting threshold length can be set according to actual conditions. For example, the preset sorting threshold length can be set to 80% of the number of word units in the second vocabulary. According to the sorting result, the word units arranged in front of the preset sorting threshold length are selected, and these word units are combined together to form the target vocabulary. If the usage frequency of the word units is the same, the word units can be sorted according to the usage time, and the word units used most recently are arranged in front.
[0052] Alternatively, the manner of determining the sorting position based on the usage information can also be to sort according to the usage time of the word units, and the word units used most frequently or most recently are arranged in front, and the word units used rarely or long time ago are arranged in back. Similarly, the preset sorting threshold length can be set according to specific application scenarios and performance requirements, such as setting the preset sorting threshold length to 70% of the number of word units in the second vocabulary. According to this sorting result, the word units arranged in front of the preset sorting threshold length are selected to form the target vocabulary.
[0053] By determining the target vocabulary, the word units with low usage frequency in the second vocabulary can be removed, the size of the vocabulary is reduced, and the inference efficiency is further improved. In some embodiments, the decoding stage can include a speculative decoding process, and the inference method can further include: Step 304: generating the intermediate word units based on the third vocabulary in the language model speculative decoding process; The third vocabulary can be a predefined vocabulary used to quickly generate intermediate word units in the speculative decoding process. The construction of the third vocabulary can be based on statistical analysis of a large number of general text data sets, and the word units with high frequency of occurrence are selected as intermediate word units. In this way, in the speculative decoding stage, the language model can only calculate within the range of the third vocabulary, further reducing the calculation amount and improving the decoding speed.
[0054] In step 304, when generating the intermediate word units based on the third vocabulary, various strategies can be used.
[0055] For example, the probability prediction mechanism of the language model can be used. The language model outputs a probability value for each word unit in the third vocabulary, indicating the possibility of the word unit as the next intermediate word unit, and then selects one or more word units with the highest probability as the intermediate word unit. For example, the top K word units with the highest probability are selected as the intermediate word units, which can increase the diversity of the guess and improve the success rate of subsequent decoding.
[0056] Alternatively, the intermediate wordpiece can also be generated in combination with context information. In the speculative decoding process, not only the probability distribution of the current time is considered, but also the context information formed by the wordpiece sequence that has been generated before. Through semantic understanding and analysis of the context, the wordpiece that matches the context better is selected as the intermediate wordpiece. For example, in a text describing the symptoms of a disease, when generating the intermediate wordpiece, the wordpiece related to the symptoms of the disease is preferred, rather than some general wordpiece irrelevant to the context.
[0057] Step 305: The third word table is a subset of the second word table or the same as the second word table.
[0058] In step 305: Wherein the third word table can be flexibly set according to actual conditions. If it is desired to focus more on commonly used or specific field wordpieces in the speculative decoding process, the third word table can be set as a subset of the second word table.
[0059] For example, in the inference of a language model in the medical field, the second word table can contain all general wordpieces related to medicine, and the third word table can only select wordpieces used in high-frequency scenarios such as clinical diagnosis and treatment plan. In this way, in the speculative decoding, the intermediate wordpiece closely related to the current task can be generated more quickly.
[0060] Alternatively, if it is desired to cover as many possible wordpieces as possible in the speculative decoding process to improve the accuracy of the guess, the third word table can be set to be the same as the second word table.
[0061] For example, when processing some open-ended text generation tasks, more vocabulary possibilities may need to be considered. At this time, making the third word table consistent with the second word table can make the range of candidate wordpieces generated in the speculative decoding stage wider.
[0062] In some embodiments, when the intermediate wordpiece is a guessed wordpiece in the speculative decoding process, Step 306: If the guessed wordpiece does not exist in the second word table, both wordpieces in the wordpiece pair formed by the guessed wordpiece and the predicted wordpiece corresponding to the guessed wordpiece are added to the second word table to update the second word table.
[0063] In step 306, the predicted wordpiece is the next possible wordpiece predicted by the model according to the current guessed wordpiece and context information.
[0064] For example, in a text inference about a historical event, the guessed wordpiece is “battle”, and the predicted wordpiece predicted by the model according to the context is “victory”. If neither “battle” nor “victory” is in the second word table, “battle” and “victory” are added to the second word table as a wordpiece pair. When adding the wordpiece pair, the two wordpieces are also appended at the end of the second vocabulary, and are respectively assigned new IDs. Assuming that the second vocabulary originally has N wordpieces, and the ID range is 1-N, then the newly added guess wordpiece can be assigned an ID of N+1, and the prediction wordpiece is assigned an ID of N+2. In this way, the updated second vocabulary contains the two missing wordpieces, and in the subsequent decoding stage, the model can process the two wordpieces to avoid errors or incomplete output due to missing wordpieces.
[0065] At the same time, the mapping table needs to be updated during the updating of the second vocabulary, recording the new mapping relationship between the guess wordpiece, the prediction wordpiece and the first vocabulary and the updated second vocabulary, so as to accurately find and use them in subsequent reasoning. In this way, the second vocabulary can be continuously improved to better adapt to various input and reasoning scenarios, and the reasoning performance and accuracy of the language model can be improved.
[0066] In some embodiments, the decoding stage includes a speculative decoding process, and the reasoning method further includes: Step 307: generating a first prediction result based on the third vocabulary in the speculative decoding process of the language model, the first prediction result including a plurality of candidate sequences of consecutive wordpieces; the third vocabulary is a subset of the second vocabulary or the same as the second vocabulary; In step 307, the prediction ability of the language model can be used to generate a plurality of candidate sequences of consecutive wordpieces in combination with the wordpieces in the third vocabulary. For example, when processing a text about tourism, the language model may, based on the current input “I want to go”, in combination with the third vocabulary, predict “seaside”, “mountain area”, “city” and the like as the next possible wordpieces, and then continue to generate a plurality of candidate sequences of consecutive wordpieces such as “I want to go to the seaside for vacation”, “I want to go to the mountain area for hiking”, “I want to go to the city for sightseeing” and the like based on these prediction results.
[0067] Step 308: verifying the first prediction result based on the first vocabulary to obtain a second prediction result corresponding to the first prediction result, and comparing each position wordpiece in the first prediction result with the corresponding wordpiece in the second prediction result; In step 308, the process of verifying the first prediction result based on the first vocabulary can be to encode each candidate sequence of consecutive wordpieces in the first prediction result according to the first vocabulary, and then input it into the complete language model for reasoning to output a new prediction result, i.e. the second prediction result.
[0068] The first word table is more comprehensive in content, which ensures that the generated result is more accurate and reasonable. For example, in a translation task, the first prediction result may be a quick translation candidate sequence, but it may not be accurate. After verification by the first word table, a more correct second prediction result can be obtained. By comparing the word units at corresponding positions in the first and second prediction results, the differences between the two results can be found.
[0069] Step 309: When the first word unit at the first position in the first prediction result does not match the second word unit at the corresponding position in the second prediction result, the second word unit at the corresponding position in the second prediction result is added to the second word table as the intermediate word unit.
[0070] In step 309, the intermediate word unit is added in a similar manner as before. The word unit is appended at the end of the second word table and assigned a new ID, and the mapping table is updated. Specifically, when comparing the first prediction result and the second prediction result, the first position of the two prediction results can be compared first, and then the word units at corresponding positions in the two prediction results can be compared in turn. If the word unit in the first prediction result is the same as the word unit in the second prediction result at a certain position, it means that the guess of the speculative decoding at that position is correct, and the next position can be compared; if they are different, it means that the guess at that position is wrong.
[0071] For example, the candidate sequence of the first prediction result is
“weather”, “sunny”, “suitable”, “travel”
“weather”, “sullen”, “unsuitable”, “travel”
[0072] In some embodiments, the initial state of the second word table is a pruned word table based on the first word table, wherein at least one subset is selected from the first word table to construct the second word table based on output statistical information generated when the language model processes the general text dataset offline.
[0073] In this embodiment, the general text dataset can be a large-scale text collection covering multiple fields, multiple language styles, and multiple topics, such as news reports, novels, academic papers, social media posts, etc. By processing the general text dataset offline using the language model, output statistical information can be obtained. The output statistical information can reflect the frequency and distribution of each word unit in the dataset.
[0074] In addition, there can be various ways to select at least one subset from the first vocabulary to construct the second vocabulary based on the output statistics.
[0075] For example, a large-scale and high-quality text dataset (e.g., SlimPajama-627B) can be used first, which should be able to represent the data distribution that the model may encounter in actual use.
[0076] Then, run the language model on the calibration dataset to obtain the probability (or frequency) of each token being sampled at the output layer. Note that here the token frequency of the model output layer is counted, that is, the probability of each token being generated when the model generates text.
[0077] Next, sort the tokens by frequency (or probability) from high to low, and then select the top K tokens (Top-K, e.g., K=32000) with the highest frequency, or the tokens whose cumulative probability reaches a predetermined threshold (e.g., 95%).
[0078] The selected tokens form an initial pruned vocabulary, and the IDs of these tokens in the original vocabulary are recorded. At the same time, a mapping relationship from the original token ID to the new vocabulary ID is saved.
[0079] By pruning the first vocabulary to obtain the second vocabulary and performing reasoning based on the second vocabulary, the computational complexity of the model can be greatly reduced.
[0080] As shown in Figure 5 The processing method provided in the embodiments of the present application comprises: Step 500: obtaining a second vocabulary, the second vocabulary being used for decoding an output layer vector to obtain an output token, wherein the second vocabulary has an initial pruning size; Step 600: obtaining a target token generated by a language model in an encoding process and / or a decoding process in response to a user input; Step 700: updating the second vocabulary based on the target token.
[0081] The processing method of the embodiments can continuously optimize the second vocabulary to better adapt to the inputs of different users and various reasoning scenarios, and further improve the performance of the language model and the user experience.
[0082] For example, as shown in Figure 8 The entire processing method can include an offline dataset pruning phase and an online prompt word dynamic pruning phase.
[0083] The first step is the offline dataset pruning stage. First, the dataset distilled from the original model is input. Then, the calibration dataset is traversed, and the sampling frequency or probability of each token (i.e., lexical unit) in the vocabulary is counted. All tokens obtained from the statistics are sorted according to their occurrence probability from high to low, forming a long-tailed distribution curve. The cumulative probability of the tokens is calculated, and a threshold is set (e.g., Top-P=95%). The set of tokens whose cumulative probability reaches this threshold is selected, thus achieving the initial pruning of the original vocabulary Vocab_0, resulting in the initial vocabulary Vocab_1, i.e., the second vocabulary. This vocabulary pruning process is performed offline.
[0084] The second step is the online prompt word dynamic trimming stage. This stage starts the model inference task and loads the initial vocabulary Vocab_1 (i.e. the second vocabulary) obtained in the first step into the inference engine.
[0085] If the input prompt or predicted word is obtained from the user, it is determined whether the target word of the input prompt or predicted word exists in the initial word list Vocab_1. If not, the target word is added to the initial word list Vocab_1. This process is repeated to continuously update and gradually stabilize the word list Vocab_1.
[0086] Therefore, the processing method in this embodiment performs offline pre-pruning of the output vocabulary of the language model based on the calibration dataset, and implements a dynamic online adaptive pruning scheme during the inference process. This greatly reduces the data storage, relocation, and computation of the output matrix while ensuring lossless model inference.
[0087] like Figure 6 As shown, in some implementations, the processing method may further include: Step 701: If the target word in the encoded word does not exist in the second vocabulary, Step 702: Add the target term to the second vocabulary to update the second vocabulary. For example, such as Figure 9 As shown, according to Figure 8 After obtaining the initial vocabulary Vocab_1 (i.e. the second vocabulary), after obtaining the user's input suggestion word ID, it will determine whether the ID belongs to the initial vocabulary Vocab_1; If the input prompt word id belongs to the initial vocabulary list Vocab_1, the initial vocabulary list Vocab_1 is used for reasoning. After multiple reasonings, the low-frequency words at the end of Vocab_1 are removed to form the target vocabulary list Vocab_2. Subsequent reasoning is then performed based on the target vocabulary list Vocab_2 to accelerate the reasoning process.
[0088] If the input prompt id does not belong to the initial vocabulary Vocab_1, the initial vocabulary Vocab_0 before pruning is used for reasoning, and then it is judged whether the generated word id after reasoning belongs to the initial vocabulary Vocab_1. If it belongs to the initial vocabulary Vocab_1, the reasoning continues. If it does not belong to the initial vocabulary Vocab_1, the generated word id can be added to the initial vocabulary Vocab_1 to form a target vocabulary Vocab_2, and subsequent reasoning is carried out based on the formed target vocabulary Vocab_2, thereby speeding up the reasoning process.
[0089] Alternatively, if the input prompt id does not belong to the initial vocabulary Vocab_1, the input prompt id can be directly added to the initial vocabulary Vocab_1 to form a target vocabulary Vocab_2, and subsequent reasoning is carried out based on the formed target vocabulary Vocab_2, thereby speeding up the reasoning process.
[0090] As shown in Figure 7 , in some implementations, the processing method can further include: Step 703: obtaining an intermediate word element generated by the decoding stage reasoning process, Step 704: if the intermediate word element does not exist in the second vocabulary, Step 705: adding the intermediate word element to the second vocabulary to update the second vocabulary.
[0091] For example, as shown in Figure 10 , according to Figure 8 , after obtaining the initial vocabulary Vocab_1, in the process of speculative decoding, the prediction ability of the language model can be used through the draft model, and the candidate sequence of multiple continuous word elements can be predicted in combination with the word elements in the third vocabulary, and then it is judged whether the predicted word id in the candidate sequence belongs to the initial vocabulary Vocab_1.
[0092] If the predicted word id belongs to the initial vocabulary Vocab_1, subsequent prediction is continued based on the initial vocabulary Vocab_1. If the predicted word id does not belong to the initial vocabulary Vocab_1, the predicted word id is added to the initial vocabulary Vocab_1 to form a target vocabulary Vocab_2, and reasoning is carried out based on the target vocabulary Vocab_2 in subsequent reasoning.
[0093] The embodiments of the present application also provide an electronic device, which can include a memory and a processor.
[0094] The memory is configured to store a language model, and the processor is configured to obtain a user input prompt.
[0095] In the encoding stage of the language model, the prompt word is processed based on the first vocabulary to generate an encoded token; In the decoding output stage of the language model, a decoding output token is obtained based on the second vocabulary, Wherein the second vocabulary is a subset of the first vocabulary.
[0096] The memory can be random access memory (RAM), read-only memory (ROM), etc., which can quickly store and read data to ensure that the language model can run efficiently.
[0097] The processor can be a central processing unit (CPU), a graphics processing unit (GPU), or a tensor processing unit (TPU), etc., which has strong computing power and can quickly process the prompt word input by the user. When the processor obtains the prompt word input by the user, it will send the prompt word to the language model stored in the memory for encoding processing.
[0098] At the same time, the electronic device can also be equipped with an input and output interface for receiving the input prompt word of the user and displaying the output result generated by the language model to the user. The input interface can be a keyboard, a mouse, a microphone, etc., and the output interface can be a display screen, a loudspeaker, etc.
[0099] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or can be advantageous.
[0100] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be mutually referred to. Each embodiment focuses on the different places from other embodiments.
Claims
1. An inference method, comprising: obtaining a user input prompt word; processing the prompt word based on a first vocabulary in an encoding stage of a language model to generate an encoded token; obtaining a decoded output token based on a second vocabulary in a decoding stage of the language model; wherein the second vocabulary is a subset of the first vocabulary.
2. The inference method of claim 1, further comprising: if a target token in the encoded token does not exist in the second vocabulary; adding the target token to the second vocabulary to update the second vocabulary.
3. The inference method of claim 1, further comprising: obtaining an intermediate token generated by the decoding stage inference process; if the intermediate token does not exist in the second vocabulary; adding the intermediate token to the second vocabulary to update the second vocabulary.
4. The inference method of claim 2 or 3, further comprising: acquiring usage information of each token in the updated second vocabulary in a multiple-session inference process of the language model; determining a ranking position based on the usage information, and selecting tokens of a preset ranking threshold length to form a target vocabulary; wherein the number of tokens in the target vocabulary is less than the number of tokens in the second vocabulary.
5. The inference method of claim 3, wherein the decoding stage comprises a speculative decoding process, and the inference method further comprises: generating the intermediate token based on a third vocabulary in a speculative decoding process of the language model, wherein the third vocabulary is a subset of the second vocabulary or identical to the second vocabulary.
6. The inference method of claim 3, wherein the intermediate token is a guessed token in the speculative decoding process; if the guessed token does not exist in the second vocabulary, adding both the guessed token and a predicted token corresponding to the guessed token to the second vocabulary to update the second vocabulary.
7. The inference method of claim 6, wherein the decoding stage comprises a speculative decoding process, and the inference method further comprises: generating a first prediction result based on a third vocabulary in a speculative decoding process of the language model, the prediction result comprising a candidate sequence of multiple consecutive tokens; wherein the third vocabulary is a subset of the second vocabulary or identical to the second vocabulary; verifying the first prediction result based on the first vocabulary to obtain a second prediction result corresponding to the first prediction result, and comparing each token at a position in the first prediction result with a corresponding token in the second prediction result; when a first token at a first position in the first prediction result does not match a second token at a corresponding position in the second prediction result, adding the second token at the corresponding position in the second prediction result as the intermediate token to the second vocabulary.
8. The inference method of claim 1, wherein an initial state of the second vocabulary is a pruned vocabulary based on the first vocabulary, wherein the second vocabulary is constructed by selecting at least one subset from the first vocabulary based on output statistical information generated when the language model is processed offline on a general text dataset.
9. A processing method, comprising: obtaining a language model to generate target tokens generated in an encoding process and / or a decoding process in response to a user input; obtaining a second vocabulary for decoding output layer vectors to obtain output tokens, wherein the second vocabulary has an initial pruning size; if a target token in the encoded tokens does not exist in the second vocabulary, adding the target token to the second vocabulary to update the second vocabulary, and / or; obtaining intermediate tokens generated by the decoding stage inference process, if the intermediate tokens do not exist in the second vocabulary, adding the intermediate tokens to the second vocabulary to update the second vocabulary.
10. An electronic device comprising: a memory configured to store a language model; a processor configured to obtain a user input prompt token, in an encoding stage of the language model, processing the prompt token based on a first vocabulary to generate encoded tokens; in a decoding output stage of the language model, obtaining decoding output tokens based on a second vocabulary, wherein the second vocabulary is a subset of the first vocabulary.