Large language model reasoning method, device and equipment based on multi-level speculation sampling

Abstract

The invention discloses a large language model reasoning method, device and equipment based on multi-level speculation sampling, and relates to the technical field of artificial intelligence, the method comprises the following steps: if a first draft matched with an input text is modified, inputting the first draft into a target level draft model generating the first draft; generating a second draft according to the first draft by using the draft model of the current level of the first draft; inputting the second draft into the draft model of the next level, and judging whether the draft model of the next level modifies the second draft or not; if the second draft is modified, returning to the step of generating the second draft; or, if the second draft is not modified, returning to the step of judging whether the second draft is modified or not until the second draft generated by the final-level draft model passes the evaluation of the original model. According to the method, step-by-step reasoning from the first-level draft model is not needed, and the generation quality of the final draft is enhanced through the draft models of multiple levels, so that the reasoning efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large language model reasoning method, device and equipment based on multi-level speculative sampling. Background Art

[0002] The reasoning of large language models usually requires the use of autoregressive sampling. The reasoning process of autoregressive sampling is quite slow and requires serial decoding token by token. Therefore, the reasoning process of large language models is often constrained by memory access speed. Generating each token requires transferring all parameters from the storage unit to the computing unit, so the memory access bandwidth becomes a serious bottleneck. Speculative sampling is a scheme that can fundamentally decode the computational memory access ratio. The draft model of the speculative sampling scheme can use the same sampling distribution as the original model.

[0003] However, existing speculative sampling schemes still suffer from low efficiency. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a large language model reasoning method, device and equipment based on multi-level speculative sampling to improve the efficiency of the speculative sampling scheme.

[0005] To achieve the above purpose, an embodiment of the present application proposes a large language model reasoning method based on multi-level speculative sampling, the method comprising the following steps:

[0006] If the first draft matching the input text has been modified, the first draft is input into a target-level draft model for generating the first draft; wherein the target-level draft model is any one-level draft model in the total draft model, the total draft model includes a plurality of draft models sequentially connected in levels, and the output of the draft model at the current level is used as the input of the draft model at the next level;

[0007] generating a second draft according to the first draft using the draft model of the level at which the first draft is currently located;

[0008] Inputting the second draft into a next-level draft model, and determining whether the next-level draft model modifies the second draft;

[0009] If the second draft has been modified, returning to the step of generating the second draft according to the first draft by using the draft model of the current level of the first draft, until the second draft generated by the last level draft model in the total draft model passes the evaluation of the original model;

[0010] If the second draft has not been modified, return to the step of inputting the second draft into the next-level draft model and determining whether the next-level draft model has made any modifications to the second draft, until the second draft generated by the last-level draft model in the total draft model passes the evaluation of the original model.

[0011] In some embodiments, before the first draft that matches the input text is modified, inputting the first draft into a target-level draft model for generating the first draft, the method further comprises the following steps:

[0012] Obtaining the first draft matching the input text;

[0013] The step of obtaining the first draft that matches the input text comprises the following steps:

[0014] Obtaining input text, and preprocessing the input text to convert it into vector data;

[0015] Retrieving the vector data in a vector database to determine whether there is a search text that matches the input text;

[0016] When the search text is found, the search text is determined as the first draft.

[0017] In some embodiments, the step of retrieving the vector data in a vector database and determining whether there is a search text matching the input text comprises the following steps:

[0018] Searching in the vector database whether there is text whose cosine similarity with the vector data reaches a preset threshold; or searching in the vector database by a matching model whether there is text matching the vector data;

[0019] If it exists, the retrieved text is determined to be the retrieved text that matches the input text.

[0020] In some embodiments, before generating a second draft according to the first draft using the draft model at the level where the first draft is currently located, the method further includes the following steps:

[0021] If the first draft has not been modified, the first draft is input into the first-level draft model in the overall draft model.

[0022] In some embodiments, the method further comprises the following steps:

[0023] The input text, the modification records of the draft models at each level to the input draft, and the output results of the draft models at each level are stored in a vector database.

[0024] In some embodiments, the method further comprises the following steps:

[0025] The first draft matching the current input text is searched in the vector database.

[0026] In some embodiments, the method further comprises the following steps:

[0027] Converting the vector corresponding to the second draft evaluated by the original model into a target text;

[0028] The target text is output.

[0029] To achieve the above object, another aspect of the embodiment of the present application proposes a large language model reasoning device based on multi-level speculative sampling, the device comprising:

[0030] a jump input unit, configured to input the first draft that matches the input text into a target-level draft model for generating the first draft if the first draft has been modified; wherein the target-level draft model is any one-level draft model in the total draft model, the total draft model includes a plurality of draft models that are sequentially connected in levels, and the output of the draft model at the current level is used as the input of the draft model at the next level;

[0031] A draft generation unit, configured to generate a second draft according to the first draft by using the draft model of the level at which the first draft is currently located;

[0032] a draft evaluation unit, configured to input the second draft into a next-level draft model, and determine whether the next-level draft model makes any modification to the second draft;

[0033] a first reasoning unit, configured to return to the step of generating a second draft according to the first draft by using the draft model of the current level of the first draft if the second draft has been modified, until the second draft generated by the last level draft model in the total draft model passes the evaluation of the original model;

[0034] The second reasoning unit is used to return to the step of inputting the second draft into the next-level draft model and judging whether the next-level draft model makes any modification to the second draft if the second draft has not been modified, until the second draft generated by the last-level draft model in the total draft model passes the evaluation of the original model.

[0035] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned large language model inference method based on multi-level speculative sampling.

[0036] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned large language model inference method based on multi-level speculative sampling.

[0037] The embodiments of the present application include at least the following beneficial effects:

[0038] The present application is as follows: if the first draft matching the input text has been modified, the first draft is input into the target level draft model for generating the first draft; wherein the target level draft model is any level draft model in the total draft model, the total draft model includes a plurality of draft models connected in sequence, and the output of the draft model of the current level is used as the input of the draft model of the next level; the draft model of the level at which the first draft is currently located is used to generate the second draft according to the first draft; the second draft is input into the draft model of the next level, and the draft model of the next level is judged. Whether the second draft is modified; if the second draft is modified, then return to the step of generating the second draft according to the first draft using the draft model of the current level of the first draft, until the second draft generated by the last-level draft model in the total draft model passes the evaluation of the original model; if the second draft is not modified, then return to the step of inputting the second draft into the next-level draft model, and judging whether the next-level draft model has modified the second draft, until the second draft generated by the last-level draft model in the total draft model passes the evaluation of the original model. For the modified first draft, the present application directly jumps to the target-level draft model for generating the first draft for reasoning, without the need to start reasoning step by step from the first-level draft model, thereby improving the reasoning efficiency; moreover, the generation quality of the final draft is enhanced by draft models of multiple levels, that is, the draft model of the next level is used to evaluate the draft generated by the draft model of the previous level, reducing the number of evaluations of the draft by the original model, thereby improving the reasoning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1A flowchart of a large language model inference method based on multi-level speculative sampling provided in an embodiment of the present application;

[0041] Figure 2 An example flow chart of step S200 provided in an embodiment of the present application;

[0042] Figure 3 An example flow chart of step S202 provided in an embodiment of the present application;

[0043] Figure 4 An example flow chart of step S600 provided in an embodiment of the present application;

[0044] Figure 5 An example flow chart of a large language model inference method based on multi-level speculative sampling provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of a large language model inference device based on multi-level speculative sampling provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of the structure of a text retrieval unit provided in an embodiment of the present application;

[0047] Figure 8 A schematic diagram of the structure of a text matching unit provided in an embodiment of the present application;

[0048] Fig. 9 A schematic diagram of the structure of the reasoning output unit provided in the embodiment of the present application;

[0049] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0051] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0052] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] Before describing the embodiments of the present application in detail, some terms and related technologies that may be involved in the embodiments of the present application are first described. Some terms and related technologies that may be involved in the embodiments of the present application are subject to the following interpretations:

[0055] Large Language Model (LLM), also known as large language model, is an artificial intelligence model designed to understand and generate human language. They usually contain billions or even trillions of parameters and can model complex language structures and patterns. These models are trained on massive amounts of text data, which may come from a variety of sources such as the Internet, books, and news, so that the model can understand diverse language styles and topics. Due to its broad training base, LLM can perform well on a variety of tasks without the need for specific training for each task. This versatility makes them very suitable for zero-shot and few-shot learning scenarios. In addition, large language models also have strong generative capabilities and can generate coherent and meaningful texts, and can even imitate a specific writing style or tone. They can also conduct conversations, answer questions, or complete other language-related tasks based on the context. They have a wide range of applications, including automatic text generation, machine translation, question-answering systems, dialogue systems, summary generation, and code generation. However, training and reasoning about large models requires huge computing resources, including a large number of GPU or TPU clusters. Reasoning about very large language models often requires some additional means such as quantization and distillation to increase the reasoning speed to reduce generation costs and improve the user experience.

[0056] Speculative sampling (English: Speculative Decoding): Speculative sampling uses two models, one is the original model, and the other is a draft model that is much smaller than the original model. The draft model is used for autoregressive serial sampling, while the original target model is used to evaluate the sampling results. During the decoding process, some tokens are relatively easy to decode, while some tokens are difficult to decode. Therefore, simple token generation can be handled by a small model, while difficult tokens are handled by a large model. The small model here can adopt the same structure as the original model, but with fewer parameters. The small model not only has a smaller amount of computation, but more importantly, it reduces the need for memory access.

[0057] Autoregressive Sampling: A standard method for generating text in language models. In autoregressive sampling, the language model generates each token in turn, and the generation of each token depends on all the tokens before it. The advantage of this method is that it is simple and easy to implement, but the disadvantage is that the inference speed is slow because the generation of each token requires recalculating all the previous tokens.

[0058] How autoregressive sampling works: In autoregressive sampling, the language model first generates the first token, then generates the second token based on this token, then generates the third token based on the first two tokens, and so on. Specifically, the model outputs a probability distribution that represents the probability of each word appearing in the current context. Then, the model randomly selects a word as the next token based on this probability distribution. This process is repeated until the entire sentence or document is generated.

[0059] Advantages of autoregressive sampling:

[0060] Simple and easy to implement: The autoregressive sampling method is relatively simple and easy to understand and implement.

[0061] High training efficiency: Since only one token is processed at a time, the parallel computing capabilities of the GPU can be fully utilized during the training process.

[0062] Disadvantages of autoregressive sampling:

[0063] Slow inference speed: Since the generation of each token requires recalculating all previous tokens, the inference speed is slow, especially when generating long texts.

[0064] Exposure bias: When generating text, the model may tend to generate tokens that have already been seen, resulting in a lack of diversity in the generated text.

[0065] Application scenarios and actual effects of autoregressive sampling: Autoregressive sampling is widely used in various language models, such as the GPT series of models. Although its inference speed is slow, it is still one of the most commonly used text generation methods due to its simplicity and efficiency. In practical applications, the inference speed can be improved by optimizing algorithms and hardware, or combined with other sampling methods such as beam search to balance generation quality and speed.

[0066] The reasoning of large language models usually requires the use of autoregressive sampling. The reasoning process of autoregressive sampling is quite slow and requires serial decoding of each token. Therefore, the reasoning process of large models is often limited by the memory access speed. Generating each token requires transferring all parameters from the storage unit to the computing unit, so the memory access bandwidth becomes a serious bottleneck. Speculative sampling is a method that can fundamentally decode the memory access ratio and ensure that the sampling distribution is exactly the same as that of the original model. The working principle of speculative sampling is:

[0067] 1. Initial generation: The draft model first generates a few words according to the traditional method to establish a preliminary context.

[0068] 2. Predicting the future: Based on the current context, the draft model tries to predict the next multiple words. This step is the core of "speculation" because it does not rely on the actual generated results, but is based on the model's predictive ability.

[0069] 3. Verification and adjustment: The original model will verify whether the predicted words are reasonable. If the predicted words are consistent with the context logic, the original model will accept these words and continue to generate them; if not, the model will fall back to a certain checkpoint and regenerate these words.

[0070] 4. Loop iteration: The above process will be repeated until a complete sentence or paragraph is generated.

[0071] However, there are three main problems in the current speculative sampling reasoning process:

[0072] 1. The effect of the draft model currently used is often quite different from that of the original model, resulting in inaccurate generation of many tokens during the decoding process. In order to ensure the real-time generation effect as much as possible, it is still necessary to make multiple corrections through the original model multiple times.

[0073] 2. Currently, speculative sampling requires the draft model to generate a draft first, and the original model to modify and evaluate the draft. However, the draft model does not make full use of the historical records of draft generation and modification, which will cause the draft model to repeatedly generate the same erroneous draft.

[0074] 3. Currently, in the speculative sampling process, the process from the generation of the draft model to the evaluation and correction of the original model is serial and does not allow skipping, resulting in the efficiency of speculative sampling not reaching the upper limit.

[0075] In order to solve the problems existing in the prior art, the embodiments of the present application provide a large language model inference method, device and equipment based on multi-level speculative sampling. The technical solution of the present application includes: if the first draft matching the input text has been modified, the first draft is input into the target level draft model for generating the first draft; wherein the target level draft model is any layer of the draft model in the total draft model, the total draft model includes a plurality of draft models connected in sequence, and the output of the draft model of the current level is used as the input of the draft model of the next level; the draft model of the level where the first draft is currently located is used to generate the second draft according to the first draft; the second draft is input into the draft model of the next level, and the next level of draft model is judged. The first draft model determines whether the second draft has been modified; if the second draft has been modified, the step of generating the second draft according to the first draft using the draft model of the current level of the first draft, until the second draft generated by the last-level draft model in the total draft model passes the evaluation of the original model; if the second draft has not been modified, the step of inputting the second draft into the next-level draft model and judging whether the next-level draft model has made modifications to the second draft, until the second draft generated by the last-level draft model in the total draft model passes the evaluation of the original model. For the modified first draft, the present application directly jumps to the target-level draft model for generating the first draft for reasoning, without the need to start reasoning step by step from the first-level draft model, thereby improving the reasoning efficiency; moreover, the generation quality of the final draft is enhanced by draft models of multiple levels, that is, the draft model of the next level is used to evaluate the draft generated by the draft model of the previous level, reducing the number of evaluations of the draft by the original model, thereby improving the reasoning efficiency.

[0076] The embodiments of the present application provide a large language model reasoning method, device and equipment based on multi-level speculative sampling, which relates to the field of artificial intelligence technology. The large language model reasoning method, device and equipment based on multi-level speculative sampling provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a knowledge extraction method, etc., but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0078] Reference Figure 1 The embodiment of the present application provides a large language model reasoning method based on multi-level speculative sampling, which may include but is not limited to steps S100 to S140 (the step numbers in the embodiment of the present application are only used to distinguish the steps and do not necessarily limit the order of the steps), as follows:

[0079] S100: If the first draft matching the input text has been modified, the first draft is input into a target-level draft model for generating the first draft; wherein the target-level draft model is any layer of a draft model in a total draft model, the total draft model includes a plurality of draft models sequentially connected in layers, and the output of the draft model at the current layer serves as the input of the draft model at the next layer.

[0080] S110: Generate a second draft according to the first draft using the draft model of the level at which the first draft is currently located.

[0081] S120: Input the second draft into a next-level draft model, and determine whether the next-level draft model makes any modification to the second draft.

[0082] S130: If the second draft has been modified, return to the step of generating the second draft according to the first draft using the draft model of the current level of the first draft, until the second draft generated by the last level draft model in the total draft model passes the evaluation of the original model.

[0083] S140: If the second draft has not been modified, return to the step of inputting the second draft into the next-level draft model and determining whether the next-level draft model has made any modification to the second draft, until the second draft generated by the last-level draft model in the total draft model passes the evaluation of the original model.

[0084] As an optional implementation, if the first draft matching the input text is modified in step S100, before inputting the first draft into the target level draft model for generating the first draft, the embodiment of the present application may further include step S200:

[0085] S200: Acquire the first draft matching the input text;

[0086] Reference Figure 2 Further, S200 obtains the first draft matching the input text, including steps S201 to S203:

[0087] S201: Obtain input text, and pre-process the input text to convert it into vector data;

[0088] S202: searching the vector data in a vector database to determine whether there is a search text matching the input text;

[0089] S203: When the search text is found, the search text is determined as the first draft.

[0090] Reference Figure 3 Furthermore, S202 searches the vector data in the vector database to determine whether there is a search text matching the input text, including steps S2021 to S2022:

[0091] S2021: Searching in the vector database whether there is text whose cosine similarity with the vector data reaches a preset threshold; or searching in the vector database through a matching model whether there is text matching the vector data;

[0092] S2022: If it exists, the retrieved text is determined to be the search text that matches the input text.

[0093] As another optional implementation, before S110 generates a second draft according to the first draft using the draft model of the current level of the first draft, the embodiment of the present application may further include step S300:

[0094] S300: If the first draft has not been modified, input the first draft into the first-level draft model in the overall draft model.

[0095] Furthermore, in order to improve the efficiency and accuracy of subsequent reasoning, the embodiment of the present application may further include a step of recording historical reasoning, specifically including step S400:

[0096] S400: storing the input text, the modification records of the draft models at each level to the input draft, and the output results of the draft models at each level into a vector database.

[0097] Furthermore, in order to improve the accuracy of the first draft by using historical reasoning records, the embodiment of the present application may further include step S500:

[0098] S500: Searching the first draft matching the current input text in the vector database.

[0099] It is understandable that the input text searched in this embodiment may be different from the input text in S100, and the current input text may be a newly input text containing other content.

[0100] Reference Figure 4 The embodiment of the present application may further include a step of outputting the text obtained by reasoning in S600, which specifically includes steps S601 to S602:

[0101] S601: converting the vector corresponding to the second draft evaluated by the original model into a target text;

[0102] S602: Output the target text.

[0103] It can be understood that when the original model does not modify the second draft output by the last-level draft model, that is, the second draft output by the last-level draft model passes the evaluation of the original model, this embodiment can use the original model to generate text corresponding to the current second draft as the target text, that is, the inferred text.

[0104] Next, the solution of the embodiment of the present application will be introduced and explained in detail with reference to specific application examples.

[0105] Reference Figure 5 , this embodiment provides an example flowchart of a large language model inference method based on multi-level speculative sampling.

[0106] Specifically, this embodiment uses multi-level speculative sampling to perform reasoning, which can solve the following three problems in the following ways:

[0107] First, to address the problem of a large gap between the draft model generation effect and the original model effect, this embodiment adds multiple levels of draft models between the draft model and the original model. The expected effect is that the simplest draft model generates the simplest token, and the larger and more complex draft model evaluates and modifies the generation of short sentences or difficult tokens that are not well grasped. The original model then performs a final evaluation and correction on the entire text. In this process, the number of times the original model is called multiple times is reduced, and the number of calls to the intermediate-sized draft model is increased.

[0108] Second, this embodiment adds records of historical answers to speculative sampling. For questions with high similarity, answers can be directly extracted as the initial generation of speculative sampling without the need to generate the simplest draft model. This allows previous generations to be reused as much as possible and reduces the repeated generation of erroneous drafts.

[0109] Third, this embodiment adds the record of the results of model modification generation at each level. For some historically similar generation, it is possible to directly match and skip the non-modification operations of the low-level draft model, and directly jump to the model where the modification action occurred in the history as the starting point. For the situation where basically no modification has occurred in the historical record, it is possible to choose to skip the evaluation and modification of this round of generation and directly enter the next round of generation.

[0110] In summary, this embodiment improves the generation quality of the draft model and the efficiency of speculative sampling by adding a multi-level draft model, recording historical evaluation and modification records of similar problems, and allowing temporary skipping of the evaluation process. It can significantly reduce the cost of large model reasoning while ensuring that the generated results are equivalent to those of the original model.

[0111] Specifically, the main technical solution of this embodiment is related to the multi-level speculative sampling reasoning part, and for a user's question, the following steps are included:

[0112] Step 1. The user inputs a text question as input text, receives the text question and pre-processes it, and converts it into a vector or matrix (i.e., vector data). During the implementation process, the user can input a question or input multiple questions at the same time for multiple calls.

[0113] Step 2. Search the vector database to see if there are similar text questions. In actual implementation, matching can be performed through cosine similarity and threshold or through matching model. If the match is successful, the answer to the similar question is taken as a draft, and a search is performed to see if it has been modified. Jump to a certain level of the draft model that generates the draft. If not, no processing is performed and the first-level draft model is entered to generate the draft.

[0114] Step 3. Continue to generate drafts until the word count condition or other termination conditions are met, such as generating a termination symbol.

[0115] Step 4. Input the generated draft into the next-level draft model for evaluation and modification. If there are locations in the draft that need to be modified, modify them and return to step 3. Record the modified location and modification level, such as the logits probability that is not satisfied during the evaluation, in the historical record.

[0116] Step 5. Repeat steps 3 and 4 until the draft generated by the last-level draft model passes the evaluation of the original model.

[0117] Step 6. Perform post-processing to convert the vector of the draft generated by the last-level draft model into text and return it to the user.

[0118] Step 7. Add all the historical records of this modification, the user's text questions, and the text output by the original model to the vector database for subsequent matching.

[0119] In summary, this embodiment includes three technical improvement points:

[0120] 1. Enhance the generation effect by allowing multiple draft models to be added during the speculative sampling process of a large language model, where the draft model can be multi-layered, which is different from the simple two-level structure of the draft-native model in the prior art.

[0121] 2. Enhance generation effect and efficiency by allowing the recording of previous draft evaluation and correction records during the speculative sampling process of large language models, where the evaluation and correction records can be stored in different forms.

[0122] 3. Improve generation efficiency by allowing some layers that are unlikely to be modified to be skipped during speculative sampling of large language models. The skipping can be continuous or related restrictions can be added to limit the skipping.

[0123] Reference Figure 6 The embodiment of the present application further provides a large language model inference device based on multi-level speculative sampling, which can implement the above-mentioned large language model inference method based on multi-level speculative sampling, and the device includes:

[0124] a jump input unit, configured to input the first draft that matches the input text into a target-level draft model for generating the first draft if the first draft has been modified; wherein the target-level draft model is any one-level draft model in the total draft model, the total draft model includes a plurality of draft models that are sequentially connected in levels, and the output of the draft model at the current level is used as the input of the draft model at the next level;

[0125] A draft generation unit, configured to generate a second draft according to the first draft by using the draft model of the level at which the first draft is currently located;

[0126] a draft evaluation unit, configured to input the second draft into a next-level draft model, and determine whether the next-level draft model makes any modification to the second draft;

[0127] a first reasoning unit, configured to return to the step of generating a second draft according to the first draft by using the draft model of the current level of the first draft if the second draft has been modified, until the second draft generated by the last level draft model in the total draft model passes the evaluation of the original model;

[0128] The second reasoning unit is used to return to the step of inputting the second draft into the next-level draft model and judging whether the next-level draft model makes any modification to the second draft if the second draft has not been modified, until the second draft generated by the last-level draft model in the total draft model passes the evaluation of the original model.

[0129] In some embodiments, the apparatus further comprises a text retrieval unit, the text retrieval unit being configured to obtain the first draft that matches the input text before inputting the first draft into the target level draft model for generating the first draft if the first draft that matches the input text is modified;

[0130] Reference Figure 7 , the text retrieval unit comprises:

[0131] A text processing unit, used for obtaining input text, preprocessing the input text and converting it into vector data;

[0132] A text matching unit, used to search the vector data in a vector database and determine whether there is a search text matching the input text;

[0133] A text determination unit is used to determine the search text as the first draft when the search text is found.

[0134] Reference Figure 8 In some embodiments, the text matching unit includes:

[0135] A text matching subunit is used to search in the vector database whether there is text whose cosine similarity with the vector data reaches a preset threshold; or to search in the vector database through a matching model whether there is text matching the vector data;

[0136] The text judgment unit is used to judge the retrieved text as the search text matching the input text if it exists.

[0137] In some embodiments, the device also includes a direct input unit, which is used to input the first draft into the first-level draft model in the total draft model before generating a second draft based on the first draft using the draft model at the level at which the first draft is currently located, if the first draft has not been modified.

[0138] In some embodiments, the device further includes an inference recording unit, which is used to store the input text, the modification records of the draft models at each level to the input draft, and the output results of the draft models at each level into a vector database.

[0139] In some embodiments, the apparatus further comprises an inference record application unit, wherein the inference record application unit is configured to search the vector database for the first draft that matches the current input text.

[0140] Reference Fig. 9 In some embodiments, the device further comprises an inference output unit, wherein the inference output unit comprises:

[0141] a vector conversion unit, configured to convert the vector corresponding to the second draft evaluated by the original model into a target text;

[0142] A text output unit is used to output the target text.

[0143] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0144] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned large language model reasoning method based on multi-level speculative sampling when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0145] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0146] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0147] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0148] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes a large language model reasoning method based on multi-level speculative sampling in the embodiment of this application;

[0149] Input / output interface 1003, used to implement information input and output;

[0150] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0151] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0152] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0153] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned large language model reasoning method based on multi-level speculative sampling.

[0154] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0155] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0156] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0157] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0158] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0160] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0161] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0162] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0163] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0165] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0166] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A large language model inference method based on multi-level speculative sampling, characterized in that: The method comprises the following steps: If the first draft matching the input text has been modified, the first draft is input into a target-level draft model for generating the first draft; wherein the target-level draft model is any one-level draft model in the total draft model, the total draft model includes a plurality of draft models sequentially connected in levels, and the output of the draft model at the current level is used as the input of the draft model at the next level; generating a second draft according to the first draft using the draft model of the level at which the first draft is currently located; Inputting the second draft into a next-level draft model, and determining whether the next-level draft model modifies the second draft; If the second draft has been modified, returning to the step of generating the second draft according to the first draft by using the draft model of the current level of the first draft, until the second draft generated by the last level draft model in the total draft model passes the evaluation of the original model; If the second draft has not been modified, return to the step of inputting the second draft into the next-level draft model and determining whether the next-level draft model has made any modifications to the second draft, until the second draft generated by the last-level draft model in the total draft model passes the evaluation of the original model.

2. The large language model inference method based on multi-level speculative sampling according to claim 1, characterized in that: Before the first draft that matches the input text is modified, inputting the first draft into the target-level draft model for generating the first draft, the method further comprises the following steps: Obtaining the first draft matching the input text; The step of obtaining the first draft that matches the input text comprises the following steps: Obtaining input text, and preprocessing the input text to convert it into vector data; Retrieving the vector data in a vector database to determine whether there is a search text that matches the input text; When the search text is found, the search text is determined as the first draft.

3. The large language model inference method based on multi-level speculative sampling according to claim 2 is characterized in that: The step of retrieving the vector data in a vector database and determining whether there is a search text matching the input text comprises the following steps: Searching in the vector database whether there is text whose cosine similarity with the vector data reaches a preset threshold; or searching in the vector database by a matching model whether there is text matching the vector data; If it exists, the retrieved text is determined to be the retrieved text that matches the input text.

4. The large language model inference method based on multi-level speculative sampling according to claim 1, characterized in that: Before generating a second draft according to the first draft by using the draft model at the level where the first draft is currently located, the method further includes the following steps: If the first draft has not been modified, the first draft is input into the first-level draft model in the overall draft model.

5. The large language model inference method based on multi-level speculative sampling according to claim 1, characterized in that: The method further comprises the following steps: The input text, the modification records of the draft models at each level to the input draft, and the output results of the draft models at each level are stored in a vector database.

6. The large language model inference method based on multi-level speculative sampling according to claim 5, characterized in that: The method further comprises the following steps: The first draft matching the current input text is searched in the vector database.

7. A large language model inference method based on multi-level speculative sampling according to any one of claims 1 to 6, characterized in that: The method further comprises the following steps: Converting the vector corresponding to the second draft evaluated by the original model into a target text; The target text is output.

8. A large language model inference device based on multi-level speculative sampling, characterized in that: The device comprises: a jump input unit, configured to input the first draft that matches the input text into a target-level draft model for generating the first draft if the first draft has been modified; wherein the target-level draft model is any one-level draft model in the total draft model, the total draft model includes a plurality of draft models that are sequentially connected in levels, and the output of the draft model at the current level is used as the input of the draft model at the next level; A draft generation unit, configured to generate a second draft according to the first draft by using the draft model of the level at which the first draft is currently located; a draft evaluation unit, configured to input the second draft into a next-level draft model, and determine whether the next-level draft model makes any modification to the second draft; a first reasoning unit, configured to return to the step of generating a second draft according to the first draft by using the draft model of the current level of the first draft if the second draft has been modified, until the second draft generated by the last level draft model in the total draft model passes the evaluation of the original model; The second reasoning unit is used to return to the step of inputting the second draft into the next-level draft model and judging whether the next-level draft model makes any modification to the second draft if the second draft has not been modified, until the second draft generated by the last-level draft model in the total draft model passes the evaluation of the original model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements a large language model inference method based on multi-level speculative sampling as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a large language model reasoning method based on multi-level speculative sampling is implemented as described in any one of claims 1 to 7.

Images