Knowledge base-based decoding method, apparatus and device

Abstract

A knowledge base-based decoding method, comprising: matching obtained generated tokens of a large language model (LLM) with tokens within a token knowledge base, so as to obtain recalled tokens; by using the LLM, verifying the recalled tokens, so as to obtain a first token that passes verification; and by using the first token, updating the generated tokens. In this way, the present application implements parallel decoding on the basis of a token knowledge base without the need of deploying two models, thus enabling simple engineering deployment and practical implementation; and initializing the token knowledge base by means of texts rewritten by an LLM enables domain knowledge tokens within the token knowledge base to better meet the requirements of the LLM, so as to improve the acceptance rate of recalled tokens, thus accelerating LLM reasoning.

Description

A decoding method, device and equipment based on a knowledge base

[0001] The present application claims priority from the Chinese patent application No. 202410867947.6 filed on June 28, 2024, and entitled "A decoding method, device and equipment based on a knowledge base", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of computers, in particular to a decoding method, device and equipment based on a knowledge base. BACKGROUND

[0003] At present, in order to improve the speed of the inference process of a large language model (LLM), two models are used for parallel decoding. One model is the original LLM, and the other model is a much smaller approximation model. The approximation model is used to generate a plurality of tokens, and the original LLM is used to verify the plurality of tokens generated by the approximation model to determine whether to accept the tokens generated by the approximation model.

[0004] However, the method of using two models for parallel decoding requires the participation of the approximation model in the decoding process, and the approximation model and the original LLM need to be aligned, which causes problems in engineering deployment and actual landing. SUMMARY

[0005] The embodiments of the present application provide a decoding method, device and equipment based on a knowledge base, which realizes parallel decoding based on a token knowledge base, so that two models do not need to be deployed, and engineering deployment and actual landing are relatively simple.

[0006] To solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0007] In a first aspect, the embodiments of the present application provide a decoding method based on a knowledge base, which comprises: matching an already generated token of a large language model (LLM) with a token in a token knowledge base to obtain a recalled token; the token knowledge base comprises: a domain knowledge token, the domain knowledge token being obtained by segmenting a rewritten text, the rewritten text being obtained by rewriting a domain knowledge text by the LLM, and the already generated token being obtained by inferring an input text by the LLM; verifying the recalled token by the LLM to obtain a first token that passes the verification; the first token is part of the recalled token; and updating the already generated token by using the first token.

[0008] In the method provided by the embodiments of the present application, parallel decoding is implemented based on the token knowledge base, so that two models do not need to be deployed, and engineering deployment and actual landing are relatively simple. Moreover, the text initialized by the LLM is rewritten to make the domain knowledge tokens in the token knowledge base more in line with the requirements of the LLM, so that more tokens that pass the verification are obtained from the recall tokens in the token knowledge base, so as to improve the acceptance rate of the recall tokens, thereby accelerating the reasoning of the LLM.

[0009] In combination with the first aspect, in a possible implementation manner of the first aspect, the token knowledge base further includes: output tokens and input tokens, the output tokens including tokens generated and / or passing the verification in the process of reasoning of the LLM, and the input tokens including tokens corresponding to the input text. In the implementation manner of the present application, the token knowledge base is constantly updated by using the input tokens and the output tokens in the process of use, so that the tokens in the token knowledge base are constantly updated according to the use of the user, the token knowledge base is more perfect, and the timeliness of the token knowledge base can be improved, so that more output in line with the requirements of the user can be obtained subsequently.

[0010] In combination with the first aspect, in a possible implementation manner of the first aspect, the method further includes: increasing the frequency of the corresponding token in the token knowledge base based on the first token; and reducing the frequency of the corresponding token in the token knowledge base based on the second token. The second token is a token that does not pass the verification in the recall token. In the implementation manner of the present application, the frequency of the token in the knowledge base is updated according to the verification result of the LLM, the frequency of the token that passes the verification is increased, and the frequency of the token that does not pass the verification is reduced, so that the token knowledge base is dynamically updated, so that the timeliness of the token knowledge base can be improved, and more output in line with the requirements of the user can be obtained subsequently.

[0011] With reference to the first aspect, in a possible implementation of the first aspect, the recall token includes N recall drafts, the recall draft includes at least one token, and the method further includes: if N is greater than a set value, obtaining M recall drafts from the N recall drafts according to corresponding frequencies of the N recall drafts in the token knowledge base; N is a positive integer, M is a positive integer, and N is greater than M; and correspondingly, verifying the recall token by using the LLM to obtain the first token that passes the verification, including: verifying the M recall drafts by using the LLM to obtain the first token. In the implementation of the present application, when the data amount of the recall token is large, M recall drafts can be reasonably selected based on the corresponding frequencies of the recall drafts, so that the acceptance rate of the recall token can be further improved, and thus the reasoning speed of the LLM can be further improved.

[0012] With reference to the first aspect, in a possible implementation of the first aspect, verifying the recall token by using the LLM to obtain the first token that passes the verification includes: merging the recall token to obtain a first to-be-verified draft; the length of the first to-be-verified draft is not greater than the longest draft length; and verifying the first to-be-verified draft by using the LLM and the generated token to obtain the first token. In the implementation of the present application, by merging the recall token, the plurality of recall drafts can be simultaneously verified, and thus the reasoning speed of the LLM can be improved.

[0013] With reference to the first aspect, in a possible implementation of the first aspect, the method further includes: adjusting the longest draft length according to the number of tokens in the first token. In the implementation of the present application, the longest draft length can be dynamically adjusted according to the number of tokens that pass the verification, so that the first to-be-verified draft with a more reasonable length can be used for verification, and redundant parallel decoding calculation can be reduced.

[0014] With reference to the first aspect, in a possible implementation of the first aspect, adjusting the longest draft length according to the number of tokens in the first token includes: obtaining an accepted length according to the number of tokens in the first token and the number of tokens in a third token; the third token is a token that passes the verification for K times before the first token is obtained, and K is a positive integer; if the accepted length is greater than a set length, adjusting the longest draft length by using the set length; and if the accepted length is less than or equal to the set length, adjusting the longest draft length by using the accepted length. In the implementation of the present application, the longest draft length can be more reasonably determined in combination with the token that passes the verification and the set length, and thus redundant parallel decoding calculation can be reduced.

[0015] With reference to the first aspect, in a possible implementation form of the first aspect, the first token comprises a first sub-token and a second sub-token, the first to-be-verified draft comprises the first sub-token and the second sub-token, and updating the generated token by using the first token comprises: if the first sub-token and the second sub-token in the first to-be-verified draft are separated by at least one token, and the first sub-token is located before the second sub-token, updating the generated token by using the first sub-token. In the implementation form of the present application, in the case that the tokens that pass the verification in the first to-be-verified draft are discontinuous, the carrying delay of the NPU for the KV cache is too large, and by retaining the continuous tokens and updating the generated token, the problem of too large carrying delay can be avoided.

[0016] With reference to the first aspect, in a possible implementation form of the first aspect, after updating the generated token by using the first token, the method further comprises: generating a second to-be-verified draft by using the second sub-token. In the implementation form of the present application, the second to-be-verified draft can be generated by using the second sub-token, so that the second sub-token that passes the verification in the front can be quickly received in the next round, thereby improving the inference speed of the LLM.

[0017] With reference to the first aspect, in a possible implementation form of the first aspect, before inferring the input text by using the LLM, the method further comprises: performing word segmentation on the input text to obtain a fourth token; and updating the token knowledge base by using the fourth token. In the implementation form of the present application, the token knowledge base will be more perfect in the process of use by updating the token knowledge base by using the token corresponding to the input text, so that the subsequent output can be more in line with the requirements of the user.

[0018] The second aspect, the present application embodiment provides a decoding device based on knowledge base, the device comprises:

[0019] The matching module is configured to match the generated identifier token of the large language model (LLM) with the token in the token knowledge base to obtain a recall token. The token knowledge base comprises: a domain knowledge token, the domain knowledge token being obtained by performing word segmentation on a rewritten text, the rewritten text being obtained by rewriting a domain knowledge text by using the LLM, and the generated token being obtained by inferring an input text by using the LLM.

[0020] The verification module is configured to verify the recall token by using the LLM to obtain a first token that passes the verification. The first token is part of the recall token.

[0021] an updating module configured to update the generated token with the first token.

[0022] The knowledge base based decoding apparatus has the function of implementing the knowledge base based decoding method in the first aspect or any possible implementation manner of the first aspect. The function can be implemented by hardware, or by executing corresponding software by hardware, and the hardware or software includes one or more modules corresponding to the above function.

[0023] The beneficial effects shown in the first aspect are similar to those shown in the first aspect, and are described in detail in the first aspect, which will not be repeated here.

[0024] In a third aspect, the embodiments of the present application provide a knowledge base based decoding device, which can include a processor and a memory, the memory and the processor are coupled, and the processor is configured to execute the knowledge base based decoding method in the first aspect or any possible implementation manner of the first aspect.

[0025] In a fourth aspect, the embodiments of the present application provide a computer program product containing instructions, which, when running on a computer, causes the computer to execute the knowledge base based decoding method in the first aspect or any possible implementation manner of the first aspect.

[0026] In a fifth aspect, the embodiments of the present application provide a computer readable storage medium, which stores instructions, when running on a computer, causes the computer to execute the knowledge base based decoding method in the first aspect or any possible implementation manner of the first aspect.

[0027] In a sixth aspect, the embodiments of the present application provide a chip system, which includes a processor for supporting the device to implement the functions involved in the above aspects, such as sending or processing the data and / or information involved in the above method. In a possible design, the chip system further includes a memory for saving the necessary program instructions and data of the device. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0028] In a seventh aspect, the embodiments of the present application provide a chip, which includes one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from the memory of an electronic device and send a signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the knowledge base based decoding method in the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0029] FIG. 1 is a schematic diagram of the architecture of a knowledge base based decoding system provided by the embodiments of the present application;

[0030] FIG. 2 is a flow block diagram of a knowledge base-based decoding method according to an embodiment of the present application;

[0031] FIG. 3 is a flow diagram of another knowledge base-based decoding method according to an embodiment of the present application;

[0032] FIG. 4 is a flow block diagram of another knowledge base-based decoding method according to an embodiment of the present application;

[0033] FIG. 5 is a schematic diagram of a token knowledge base according to an embodiment of the present application;

[0034] FIG. 6 is a schematic diagram of a verification result according to an embodiment of the present application;

[0035] FIG. 7 is a schematic diagram of parallel decoding according to an embodiment of the present application;

[0036] FIG. 8 is a flow block diagram of another knowledge base-based decoding method according to an embodiment of the present application;

[0037] FIG. 9 is a structural schematic diagram of a knowledge base-based decoding apparatus according to an embodiment of the present application;

[0038] FIG. 10 is a structural schematic diagram of a knowledge base-based decoding device according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The embodiments of the present application will be described below with reference to the accompanying drawings. It is obvious to those skilled in the art that, as technology develops and new scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0040] The terms "first", "second", and the like in the specification and claims of the present application and the above-described accompanying drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is merely a distinguishing way used in the description of the embodiments of the present application to describe the objects with the same attributes. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product, or device that includes a series of units does not have to be limited to those units, but can include other units not clearly listed or inherent to the process, method, product, or device.

[0041] Some special terms related to the embodiments of the present application are described below.

[0042] (1) Large language model (LLM)

[0043] LLM refers to those language models trained on large-scale text corpus, containing hundreds of billions (or more) of parameters, which are a series of artificial intelligence models designed to understand and generate human language. They are trained on a large amount of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc., such as GPT-4, LLaMA, etc.

[0044] The inference calculation of LLM includes a prefill stage and a decode stage. The prefill stage is to infer the first token according to the user input, and the decode stage is to generate the next token according to the first token output by the prefill stage and / or the token generated by the previous decode.

[0045] (2) Token

[0046] Token represents the smallest unit of meaning that LLM can understand and generate, and is the basic unit of the model. According to the specific tokenization scheme used, token can represent a word, a part of a word, or even only a character. Token is the language component of the model, which can be assigned a numerical value or identifier and arranged in sequence or vector, and input or output from the model.

[0047] (3) Auto-Regressive Decoding

[0048] In natural language tasks, a pre-trained large language model is usually used to generate output text according to given input text, such as generating an ending from a beginning or generating an answer from a question, etc. In order to generate output text, the large language model needs to predict each token one by one until a termination condition is reached, such as a punctuation mark or a maximum length, which is the process of auto-regressive decoding.

[0049] (4) Parallel Decoding

[0050] Parallel decoding refers to predicting multiple tokens at once, i.e., generating multiple tokens at once, which are verified by the large model in parallel.

[0051] (5) Tokenization and Tokenizer

[0052] Tokenization refers to the process of converting and splitting text into tokens, i.e., converting and splitting text into a format that the model can understand, and tokenizer refers to the model or tool used for tokenization.

[0053] (6) Neural-Networks Process Units (NPU)

[0054] NPU is designed for Internet of Things artificial intelligence, used to accelerate the operation of neural networks, solve the problem of low efficiency of traditional chips in neural network operation.

[0055] (7) KV cache

[0056] K and V in KV cache are the state values of key and value in attention mechanism, and KV cache only appears in the autoregressive decoder of the Transformer structure, which is the inference acceleration function of the Transformer.

[0057] The knowledge base-based decoding method provided by the embodiments of the present application can be applied to text continuation, text summarization, intelligent question answering, code generation, etc. This is not exhaustive in the embodiments of the present application.

[0058] Before describing the knowledge base-based decoding method provided by the embodiments of the present application in detail, a knowledge base-based decoding system architecture provided by the embodiments of the present application is described. Please refer to FIG. 1, which is a schematic diagram of one architecture of a knowledge base-based decoding system provided by the embodiments of the present application. The inference framework of LLM can receive user input, perform parallel decoding based on the token knowledge base, and obtain the output corresponding to the user input. For example, the user input can be "Please continue to write 'a cat'", and the output can be "a cat on a chair". It can be understood that the above is only an example and should not be construed as limiting the embodiments of the present application.

[0059] Specifically, please refer to FIG. 2, which is a flow block diagram of a knowledge base-based decoding method provided by the embodiments of the present application. The method can be executed by a knowledge base-based decoding device, such as a cloud server, a local server, or a personal computer. The knowledge base-based decoding method provided by the embodiments of the present application can include:

[0060] 201. According to the generated identifier token of the large language model LLM and the token in the token knowledge base, the recall token is obtained.

[0061] The token knowledge base can include: domain knowledge token, the domain knowledge token is obtained by tokenizing the rewritten text, the rewritten text is obtained by rewriting the domain knowledge text by LLM, and the generated token is obtained by LLM inference on the input text.

[0062] The generated token in the embodiments of the present application can include: a token generated in the prefill stage and / or a token generated by previous decoding, and the input text refers to the text input to the LLM.

[0063] In the embodiments of the present application, the domain knowledge text can be rewritten by the LLM to obtain the corresponding rewritten text, and then the rewritten text is segmented to obtain the domain knowledge token. In this way, the domain knowledge token is more consistent with the language expression style and output mode of the LLM while preserving the original content, which can improve the acceptance rate of subsequent recall tokens. The domain knowledge text refers to a text including corresponding domain knowledge, which can be meteorological domain knowledge text, medical domain knowledge text, computer domain knowledge text, financial domain knowledge text, etc. The embodiments of the present application do not limit this, and the corresponding domain knowledge text can be selected according to actual needs to obtain the corresponding domain knowledge token by rewriting and segmentation, and to construct the corresponding token knowledge base.

[0064] The process of rewriting the domain knowledge text by the LLM can include: inputting the domain knowledge text into the LLM and giving corresponding LLM instructions, such as "rewrite the input text without changing the original meaning", and then obtaining the rewritten text output by the LLM. For example, the domain knowledge text is "today is a sunny day", and the rewritten text obtained by rewriting the domain knowledge text by the LLM is "today is a big sunny day". It can be understood that the above is only an exemplary description and should not be understood as a limitation of the embodiments of the present application.

[0065] It should be noted that in actual application, since the user's domain knowledge text may need to be kept secret, the user can construct and maintain the token knowledge base by himself through the construction tool provided by the manufacturer. For domain knowledge text that does not need to be kept secret, the manufacturer can also assist the user to construct and maintain the token knowledge base.

[0066] The matching in the embodiments of the present application can be prefix matching, and the last i tokens of the generated token can be matched with the tokens in the token knowledge base. If there is a draft in the domain knowledge base that is completely identical to the last i tokens of the generated token, the matching is successful, and then the last j tokens corresponding to the draft with successful matching can be recalled to obtain the recalled token. Wherein, i and j are positive integers. For example, the last 1 token of the generated token is "sits", and there is a draft that is completely identical to "sits", which is "...sits on my knee..." and "...sits on a chair...". The last 3 tokens corresponding to the draft with successful matching are recalled to obtain the recalled token, which is "on my knee" and "on a chair". It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0067] 202. Verify the recalled token by using the LLM to obtain a first token that passes the verification.

[0068] The first token is part of the recalled token. For example, the recalled token is "on my knee" and "on a chair", and the first token is "on a chair". It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0069] In the embodiments of the present application, the multiple tokens generated by the LLM this time can be used to compare and verify the recalled token. If at least one token in the generated multiple tokens is identical to at least one token in the recalled token, the identical at least one token is the first token that passes the verification.

[0070] 203. Update the generated token by using the first token.

[0071] In the embodiments of the present application, the first token can be added to the end of the generated token to update the generated token. For example, the generated token is "sits", and after updating the generated token by using the first token, the generated token is "sits on a chair". It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0072] It should be noted that the above 201 to 203 can be performed in the decode stage of LLM inference.

[0073] It can be seen that in the method provided in the embodiments of the present application, parallel decoding is realized based on the token knowledge base, so that two models do not need to be deployed, and engineering deployment and actual landing are relatively simple. Moreover, the token knowledge base is initialized by the text rewritten by the LLM, so that the domain knowledge tokens in the token knowledge base are more in line with the requirements of the LLM, and more tokens that pass the verification are obtained from the recall tokens in the token knowledge base, so that the acceptance rate of the recall tokens can be improved, thereby accelerating the reasoning of the LLM.

[0074] Please refer to FIG. 3, which is a flowchart of another knowledge base-based decoding method provided in the embodiments of the present application. The another knowledge base-based decoding method will be described in combination with FIG. 3. The method can include: in the initialization stage, performing offline rewriting on the domain knowledge text by the LLM to obtain rewritten text, then performing tokenization on the rewritten text by the tokenizer to obtain domain knowledge tokens, and storing the domain knowledge tokens in the token knowledge base. In the LLM reasoning before stage, performing tokenization on the user input by the tokenizer to obtain input tokens, and storing the input tokens in the token knowledge base. In the prefill stage of the LLM reasoning, generating a first token by using the input tokens. In the decode stage of the LLM reasoning, matching the generated tokens with the tokens in the token knowledge base to obtain recall tokens, the generated tokens can include the first token generated in the prefill stage and / or tokens that pass the verification; the LLM performs parallel decoding and verification based on the recall tokens to obtain tokens that pass the verification; the tokens that pass the verification are used to update the generated tokens, and the token knowledge base is updated; when the end condition is not reached, returning to perform the matching of the generated tokens with the tokens in the token knowledge base to obtain the recall tokens; when the end condition is reached, obtaining the corresponding output.

[0075] Please refer to FIG. 4, which is a flowchart of another knowledge base-based decoding method provided in the embodiments of the present application. The another knowledge base-based decoding method will be described in combination with FIG. 4. The method can include:

[0076] 401. Tokenizing the input text to obtain a fourth token.

[0077] The input text refers to the text input into the LLM, and the tokenizer can be used to tokenize the input text in the embodiments of the present application.

[0078] 402. Updating the token knowledge base by using the fourth token.

[0079] It can be understood that the token knowledge base in the embodiments of the present application is similar to the token knowledge base in the above-mentioned embodiments, and details are not repeated here for the same parts.

[0080] Please refer to FIG. 5, which is a schematic diagram of a token knowledge base provided by an embodiment of the present application. The token knowledge base can include domain knowledge tokens, output tokens and input tokens, which are stored separately, for example, the domain knowledge tokens are stored in a first storage area, the output tokens are stored in a second storage area, and the input tokens are stored in a third storage area. The output tokens include tokens generated and / or verified in the inference process of the LLM. The input tokens can include tokens corresponding to the input text. In the embodiments of the present application, the token knowledge base is initialized by the domain knowledge tokens. In the process of use, the token knowledge base is constantly updated by the input tokens and the output tokens. In this way, the tokens in the token knowledge base are constantly updated according to the use of the user, which can improve the timeliness of the knowledge base and make the subsequent output more in line with the requirements of the user.

[0081] The token knowledge base in the embodiments of the present application can be a prefix tree structure, which includes nodes and edges. The input tokens, the output tokens and the domain knowledge tokens are distinguished by the type attributes of the nodes and the edges. The edges also contain frequency attributes. If the same text appears multiple times, the frequency of the corresponding edge is increased. For example, the frequency of my after on is 3, and the frequency of chair is 2, etc. It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0082] In the embodiments of the present application, updating the token knowledge base by the fourth token can include: if the input token in the token knowledge base exists, the frequency of the token corresponding to the input token is increased; if the input token in the token knowledge base does not exist, the fourth token is added to the input token in the token knowledge base in the form of a trie tree. The frequencies of the input tokens corresponding to different users in the same batch in the token knowledge base can be calculated independently, so as to avoid the interference of other users' input.

[0083] The inference result of the LLM can have the same part as the input text. In order to better utilize the overlapping part between the input text and the inference result, the input tokens in the token knowledge base can be emptied after obtaining the inference result of the LLM each time.

[0084] 403. The generated tokens of the LLM are matched with the tokens in the token knowledge base to obtain recall tokens.

[0085] It can be understood that 403 is similar to 201 described above, and the same part will not be described here.

[0086] In the embodiments of the present application, the generated tokens of the LLM can be used to simultaneously retrieve the domain knowledge tokens, the output tokens and the input tokens in the token knowledge base, and match them with the tokens in the token knowledge base to obtain the recall tokens. In the case that the recall tokens include too many recall drafts, if at least one recall draft in the recall tokens is from the domain knowledge tokens, at least one recall draft is from the output tokens, and at least one recall draft is from the input tokens, at least one recall draft from the domain knowledge tokens, at least one recall draft from the output tokens and at least one recall draft from the input tokens are retained for subsequent verification.

[0087] 404. The recall tokens are merged to obtain a first draft to be verified.

[0088] The length of the first draft to be verified is not more than the longest draft length. The longest draft length can be used to limit the number of tokens that can be verified at a time during parallel decoding.

[0089] In the embodiments of the present application, the multiple recall drafts in the recall tokens can be merged into a tree structure draft, and the first draft to be verified can be obtained based on the tree structure draft.

[0090] 405. The first draft to be verified is verified by using the LLM and the generated tokens to obtain first tokens.

[0091] In the embodiments of the present application, the generated tokens and the first draft to be verified can be input into the LLM to generate multiple tokens in parallel to obtain generated tokens. Then, the first draft to be verified is verified by using the generated tokens, and the first tokens are obtained according to the tokens in the first draft to be verified that are the same as the tokens in the generated tokens.

[0092] Please refer to FIG. 6 and FIG. 7, FIG. 6 is a schematic diagram of parallel decoding provided by an embodiment of the present application, and FIG. 7 is a schematic diagram of a verification result provided by an embodiment of the present application. The parallel decoding provided by an embodiment of the present application will be exemplarily described in combination with FIG. 6 and FIG. 7. In FIG. 6, "a cat" corresponds to an input token, the first token of the output in the prefill stage is "sits", that is, a generated token, the generated token "sits" is matched with the tokens in the token knowledge base, and recall tokens including three recall drafts are obtained, the three recall drafts "on my knee", "on a table", and "on a chair" are combined into a tree structure draft, and the first to-be-verified draft "on my knee a chair table" is obtained based on the tree structure draft. The first to-be-verified draft "on my knee a chair table" is spliced at the end of the generated token "sits", and is input into the LLM for parallel decoding and verification. The generated token in FIG. 7 is a plurality of tokens generated by this parallel decoding, and "on", "a", and "chair" in the generated token are the same as "on", "a", and "chair" in the first to-be-verified draft, so that the first token is "on a chair". In this way, the plurality of recall drafts are simultaneously verified by combining the recall tokens, and the speed of LLM inference is improved. It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0093] Optionally, the recall token in the embodiment of the present application can include N recall drafts, each recall draft can include at least one token, if N is greater than a set value, M recall drafts are obtained from the N recall drafts according to the corresponding frequency of the N recall drafts in the token knowledge base; N is a positive integer, M is a positive integer, and N is greater than M; accordingly, the M recall drafts can be combined to obtain a first to-be-verified draft, the first to-be-verified draft is verified by the LLM to obtain a first token. In this way, when the data amount of the recall token is large, M recall drafts can be reasonably selected based on the corresponding frequency of the recall drafts, so that the acceptance rate of the recall token can be further improved, and the inference speed of the LLM can be further improved.

[0094] Optionally, the token generated by each parallel decoding and / or the token passed by each verification can be updated to the output token in the token knowledge base in the embodiments of the present application to further improve the token knowledge base. If the token generated by parallel decoding and / or the token passed by verification already exists in the output token in the token knowledge base, the frequency of the token corresponding to the output token is increased, and if the token generated by parallel decoding and / or the token passed by verification does not exist in the output token in the token knowledge base, the token generated by parallel decoding and / or the token passed by verification is added to the output token.

[0095] 406, updating the generated token by using the first token.

[0096] It can be understood that 406 is the same as 203 described above, and therefore will not be described again.

[0097] 407, increasing the frequency of the corresponding token in the token knowledge base based on the first token.

[0098] In the embodiments of the present application, the frequency of the token passed by verification can be increased by 1 in the token knowledge base. It can be understood that the specific increase number of the frequency is not limited in the embodiments of the present application, and can be set according to actual conditions.

[0099] 408, reducing the frequency of the corresponding token in the token knowledge base based on the second token.

[0100] The second token is the token that is not passed by verification in the recall token.

[0101] In the embodiments of the present application, the frequency of the token that is not passed by verification can be reduced by 1 / n in the token knowledge base. The n represents the length of the first draft to be verified. It can be understood that the specific reduction number of the frequency is not limited in the embodiments of the present application, and can be set according to actual conditions.

[0102] Please continue to refer to FIG. 7, and the knowledge base frequency updating method provided by the embodiment of the present application is exemplarily described in combination with FIG. 7. The tokens verified as passing in FIG. 7 are "on", "a", and "chair", and the tokens verified as failing are "my", "knee", and "table". The frequencies corresponding to "on", "a", and "chair" in the token knowledge base can be respectively increased by 1, and the frequencies corresponding to "my", "knee", and "table" in the token knowledge base can be respectively reduced by 1 / 6. In this way, the frequencies of the tokens in the knowledge base are updated according to the verification results of the LLM, the frequencies of the tokens verified as passing are increased, and the frequencies of the tokens verified as failing are reduced, so as to dynamically update the token knowledge base, improve the timeliness of the token knowledge base, and thus make the subsequent output more in line with the requirements of the user. It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0103] 409. Adjust the maximum length of the draft according to the number of tokens in the first token; return to execute 403.

[0104] The current maximum length of the draft can be min(max value of tokens verified as passing in the previous K+1 decoding*m, max_length). m and K are positive integers, m can be 2, 3, etc., K can be 1, 2, etc., and max_length represents a set length, which can be adaptively set according to actual experience. It can be understood that when there is no token verified as passing at the beginning of the inference of the LLM, the set length is directly used as the maximum length of the draft.

[0105] In the embodiment of the present application, the acceptance length can be obtained according to the number of tokens in the first token and the number of tokens in the third token; if the acceptance length is greater than the set length, the maximum length of the draft is adjusted using the set length; if the acceptance length is less than or equal to the set length, the maximum length of the draft is adjusted using the acceptance length. The third token is the token verified as passing K times before the first token, K is a positive integer, and the set length can be adaptively set according to actual experience.

[0106] For example, K is 3, but only two verifications have been performed before the first token, and there are only two tokens that pass the verification, so the acceptance length is obtained by taking the token quantity of the two tokens that pass the verification before the first token and the token quantity of the first token. Assuming that the token quantity of the first token is 3, the token quantities of the two tokens that pass the verification before the first token are 2 and 4 respectively, and the set length is 10, the acceptance length is 4*2=8, which is less than 10, and the longest draft length is adjusted to 8. In this way, the longest draft length can be dynamically adjusted according to the number of tokens that pass the verification, so that subsequent verifications can be performed using a first draft with a more reasonable length, reducing redundant parallel decoding calculations. It can be understood that the above is only an example and should not be construed as limiting the embodiments of the present application.

[0107] 410. Until the end condition is reached, the output text is obtained based on the generated token.

[0108] The end condition can be the generation of a termination token or the reaching of an output length limit, etc.

[0109] As can be seen, in the method provided by the embodiments of the present application, the token knowledge base is constantly updated using input tokens and output tokens during use. In this way, the tokens in the token knowledge base are constantly updated according to the user's usage, the token knowledge base is more complete, and the frequency of tokens in the knowledge base is updated according to the verification result of the LLM. By increasing the frequency of tokens that pass the verification and reducing the frequency of tokens that fail the verification, the token knowledge base is dynamically updated. In this way, the timeliness of the token knowledge base can be improved, so that subsequent outputs that meet the user's requirements can be obtained. Moreover, the longest draft length can be dynamically adjusted according to the number of tokens that pass the verification, so that subsequent verifications can be performed using a first draft with a more reasonable length, reducing redundant parallel decoding calculations.

[0110] Please refer to the figure, FIG. 8 is a flow block diagram of another knowledge base-based decoding method provided by the embodiments of the present application. The knowledge base-based decoding method provided by the embodiments of the present application can include:

[0111] 801. The generated token of the LLM is matched with the token in the token knowledge base to obtain a recall token.

[0112] It can be understood that 801 is the same as 403 described above, and the same parts will not be described here.

[0113] 802, merge the recall tokens to obtain a first to-be-verified draft.

[0114] It can be understood that 802 is the same as 404 described above, and the same parts will not be repeated here.

[0115] 803, verify the first to-be-verified draft by using the LLM and the generated token to obtain a first token.

[0116] The first token includes a first subtoken and a second subtoken, and the first to-be-verified draft also includes the first subtoken and the second subtoken. The first subtoken refers to a token that is received in the token that passes the verification, and the second subtoken refers to a token that is not received in the token that passes the verification.

[0117] It can be understood that 803 is the same as 405 described above, and the same parts will not be repeated here.

[0118] 804, if the interval between the first subtoken and the second subtoken in the first to-be-verified draft is at least one token, and the first subtoken is located before the second subtoken, update the generated token by using the first subtoken.

[0119] It should be noted that the carrying delay of some NPU for the KV cache in the NZ format is too large, so it is necessary to use continuous tokens to update the generated token.

[0120] Please continue to refer to FIG. 7, and the updating of the generated token provided by the embodiments of the present application is exemplarily illustrated in combination with FIG. 7. In FIG. 7, the interval between the first subtoken “on” and the second subtoken “a chair” in the first to-be-verified draft is two tokens “my knee”, and “on” is located before “a chair”. The generated token “sits” is updated by using “on” to obtain the updated generated token “sits on”. In this way, in the case that the tokens that pass the verification in the first to-be-verified draft are not continuous, part of the tokens that pass the verification are discarded, and the generated token is updated by using continuous tokens. In this way, the problem of too large carrying delay of the NPU for the KV cache can be avoided. It can be understood that the above is only an exemplary illustration, and should not be understood as a limitation on the embodiments of the present application.

[0121] 805, increase the frequency of the corresponding token in the token knowledge base based on the first subtoken.

[0122] The frequency corresponding to the first sub-token can be increased by 1 in the token knowledge base in the embodiment of the application. It can be understood that the specific increase number of the frequency is not limited in the embodiment of the application, and can be specifically set according to actual conditions.

[0123] 806, decrease the frequency of the corresponding token in the token knowledge base based on the second token.

[0124] The second token is a token that does not pass verification among the recalled tokens.

[0125] It can be understood that 806 is the same as 408 in the above-mentioned embodiment, and therefore the same part will not be described again.

[0126] Please continue to refer to FIG. 7, and the another knowledge base frequency updating method provided by the embodiment of the application will be exemplarily explained in combination with FIG. 7. The first sub-token that passes verification in FIG. 7 is “on”, the second sub-token is “a chair”, and the second token is “my”, “knee” and “table”. The frequency corresponding to “on” can be increased by 1 in the token knowledge base, the frequencies of “my”, “knee” and “table” can be respectively decreased by 1 / 6 in the token knowledge base, and the frequency of “a chair” does not change. It can be understood that the above-mentioned is only exemplarily explained, and should not be understood as a limitation on the embodiment of the application.

[0127] 807, adjust the maximum draft length according to the number of tokens in the first sub-token.

[0128] The current maximum draft length can be min(max_value of tokens received in the first K+1 decoding, max_length). m and K are positive integers, for example, m can be 2, 3, etc., for example, K can be 1, 2, etc., and max_length represents a set length, which can be adaptively set according to actual experience. It can be understood that when the LLM starts reasoning, there is no received token, and the set length is directly used as the maximum draft length.

[0129] In the embodiment of the application, the acceptance length can be obtained according to the number of tokens in the first sub-token and the number of tokens in the third sub-token; if the acceptance length is greater than the set length, the maximum draft length is adjusted using the set length; if the acceptance length is less than or equal to the set length, the maximum draft length is adjusted using the acceptance length. The third token is the token received K times before the first sub-token, K is a positive integer, and the set length can be adaptively set according to actual experience.

[0130] 808, generate a second to-be-verified draft using the second sub-token.

[0131] In the embodiments of the present application, the second sub-token can be directly used as the next draft. Specifically, the second sub-token can be directly added in front of the first to-be-verified draft generated next time to generate a second to-be-verified draft, and the second to-be-verified draft is used for next verification, so that the second sub-token that passes the verification can be quickly received in the next round while avoiding too large a carrying delay, and the inference speed of the LLM is improved.

[0132] It can be seen that, in the method provided by the embodiments of the present application, by retaining the continuous token and updating the generated token, the problem of too large NPU carrying delay can be avoided. Moreover, according to the receiving result of the LLM, the frequency of the token in the knowledge base is updated, the frequency of the token that has been received is increased, and the frequency of the token that fails the verification is reduced, the token knowledge base is dynamically updated, so that the timeliness of the token knowledge base is improved, and thus the subsequent output can be more in line with the requirements of the user. Furthermore, the length of the longest draft can be dynamically adjusted according to the number of tokens that have been accepted, so that the first to-be-verified draft with a more reasonable length can be used for verification in the subsequent, and the redundant parallel decoding calculation is reduced.

[0133] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0134] In order to better implement the above-mentioned scheme of the embodiments of the present application, a related device for implementing the above-mentioned scheme is also provided below.

[0135] Please refer to FIG. 9, the decoding device 900 based on the knowledge base provided by the embodiments of the present application can include:

[0136] The matching module 901 is configured to match the generated identifier token of the large language model (LLM) with the token in the token knowledge base to obtain a recall token. The token knowledge base includes a domain knowledge token, the domain knowledge token is obtained by tokenizing a rewritten text, the rewritten text is obtained by rewriting a domain knowledge text by the LLM, and the generated token is obtained by the LLM inferring an input text.

[0137] The verification module 902 is configured to verify the recall token by using the LLM to obtain a first token that passes the verification; the first token is part of the recall token.

[0138] The updating module 903 is configured to update the generated token by using the first token.

[0139] It can be seen that, in the method provided in the embodiments of the present application, parallel decoding is implemented based on the token knowledge base, so that two models do not need to be deployed, and engineering deployment and actual landing are relatively simple. Moreover, the token knowledge base is initialized by the text modified by the LLM, so that the domain knowledge token in the token knowledge base is more in line with the requirements of the LLM, and more tokens that pass the verification are obtained from the recall token in the token knowledge base, so that the acceptance rate of the recall token can be improved, and the inference of the LLM can be accelerated.

[0140] In some embodiments of the present application, the token knowledge base further includes: output tokens and input tokens, the output tokens include tokens generated and / or passing the verification in the process of inference of the LLM, and the input tokens include tokens corresponding to the input text.

[0141] In some embodiments of the present application, the decoding device based on the knowledge base 900 further includes:

[0142] The increasing module is configured to increase the frequency of the corresponding token in the token knowledge base based on the first token.

[0143] The decreasing module is configured to decrease the frequency of the corresponding token in the token knowledge base based on the second token; the second token is a token that does not pass the verification in the recall token.

[0144] In some embodiments of the present application, the recall token includes N recall drafts, the recall draft includes at least one token, and the decoding device based on the knowledge base 900 further includes:

[0145] The obtaining module is configured to, if N is greater than a set value, obtain M recall drafts from the N recall drafts according to the frequencies of the N recall drafts in the token knowledge base; N is a positive integer, M is a positive integer, and N is greater than M.

[0146] The verification module 902 is specifically configured to verify the M recall drafts by using the LLM to obtain the first token.

[0147] In some embodiments of the present application, the verification module 902 includes:

[0148] The merging unit is configured to merge the recall tokens to obtain a first to-be-verified draft, and a length of the first to-be-verified draft is not more than the maximum draft length.

[0149] The verification unit is configured to verify the first to-be-verified draft by using the LLM and the generated token to obtain a first token.

[0150] In some embodiments of the present application, the knowledge base-based decoding apparatus 900 further includes:

[0151] The adjusting module is configured to adjust the maximum draft length according to a number of tokens in the first token.

[0152] In some embodiments of the present application, the adjusting module includes:

[0153] The obtaining unit is configured to obtain an acceptance length according to the number of tokens in the first token and a number of tokens in a third token, the third token being a token that has passed K times of verification before the first token is obtained, K being a positive integer.

[0154] The adjusting unit is configured to adjust the maximum draft length by using a set length if the acceptance length is greater than the set length.

[0155] The adjusting unit is further configured to adjust the maximum draft length by using the acceptance length if the acceptance length is less than or equal to the set length.

[0156] In some embodiments of the present application, the first token includes a first subtoken and a second subtoken, and the first to-be-verified draft includes the first subtoken and the second subtoken, and the updating module 903 is specifically configured to update the generated token by using the first subtoken if there is at least one token between the first subtoken and the second subtoken in the first to-be-verified draft, and the first subtoken is located before the second subtoken.

[0157] In some embodiments of the present application, the knowledge base-based decoding apparatus 900 further includes:

[0158] The generating module is configured to generate a second to-be-verified draft by using the second subtoken.

[0159] In some embodiments of the present application, the knowledge base-based decoding apparatus 900 further includes:

[0160] The word segmentation module is configured to perform word segmentation on the input text to obtain a fourth token.

[0161] The updating module 903 is further configured to update the token knowledge base by using the fourth token.

[0162] It should be noted that the information interaction, execution process and the like between the modules / units of the apparatus are based on the same concept as the method embodiments of the present application, and the technical effects brought by the same are the same as the method embodiments of the present application. For details, refer to the description in the foregoing method embodiments of the present application, which will not be repeated here.

[0163] Next, a knowledge base-based decoding device provided by an embodiment of the present application is introduced. Referring to FIG. 10, FIG. 10 is a structural schematic diagram of a knowledge base-based decoding device provided by an embodiment of the present application. The knowledge base-based decoding device 1000 can include one or more processors 1001 and memories 1005.

[0164] The memory 1005 can include read-only memory and random access memory, and provide the processor 1001 with instructions and data. A part of the memory 1005 can also include a non-volatile random access memory (NVRAM). The memory 1005 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or expanded sets thereof, wherein the operation instructions can include various operation instructions for implementing various operations.

[0165] The processor 1001 controls the operation of the device. In a specific application, the various components of the knowledge base-based decoding device 1000 are coupled together through a bus system, which can include a data bus, a power supply bus, a control bus and a state signal bus, etc. However, for the sake of clarity, all kinds of buses are referred to as a bus system in the figure.

[0166] The knowledge base based decoding method disclosed in the embodiments of the present application can be applied to the processor 1001 or implemented by the processor 1001. The processor 1001 can be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit or the instruction in the form of software in the processor 1001. The processor 1001 disclosed above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The processor 1001 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, or other mature storage medium in the art. The storage medium is located in the storage 1005, and the processor 1001 reads the information in the storage 1005 and combines the hardware to complete the steps of the above knowledge base based decoding.

[0167] The knowledge base based decoding device 1000 can further include one or more power supplies 1002, one or more wired or wireless network interfaces 1003, one or more input and output interfaces 1004, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0168] In the embodiments of the present application, the processor 1001 is configured to execute the knowledge base based decoding method in the embodiments corresponding to FIG. 2, FIG. 3, FIG. 4 or FIG. 8. It should be noted that the specific manner in which the processor 1001 executes each of the above steps is based on the same concept as each of the method embodiments corresponding to FIG. 1 to FIG. 8 of the present application, and the resulting technical effects are the same as each of the method embodiments corresponding to FIG. 1 to FIG. 8 of the present application. For specific content, please refer to the description of the method embodiments disclosed in the foregoing embodiments of the present application, which will not be described here.

[0169] The embodiment of the present application also provides a computer readable storage medium, which stores instructions, and when the instructions are executed on a computer, the computer executes the knowledge base based decoding method described in the foregoing embodiment shown in FIG. 2, FIG. 3, FIG. 4 or FIG. 8.

[0170] The embodiment of the present application also provides a computer program product, which comprises a program, and when the program is executed on a computer, the computer executes the knowledge base based decoding method described in the foregoing embodiment shown in FIG. 2, FIG. 3, FIG. 4 or FIG. 8.

[0171] Any processor mentioned in the foregoing can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling execution of programs of the method of the first aspect.

[0172] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0173] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CLUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0174] In the above embodiments, the implementation can be wholly or partially by software, hardware, firmware, or any combination thereof. When implemented by software, the implementation can be wholly or partially in the form of a computer program product.

[0175] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A knowledge base-based decoding method, characterized in that, The method includes: The generated identifier tokens of the Large Language Model (LLM) are matched with tokens in the token knowledge base to obtain recall tokens. The token knowledge base includes: domain knowledge tokens, which are obtained by segmenting the rewritten text. The rewritten text is obtained by rewriting the domain knowledge text using the LLM. The generated tokens are obtained by reasoning from the input text by the LLM. The recall token is verified using the LLM to obtain a first token that passes verification; the first token is a part of the recall token. Update the generated token using the first token.

2. The method according to claim 1, characterized in that, The token knowledge base also includes: output tokens and input tokens, wherein the output tokens include tokens generated and / or verified during the inference process of the LLM, and the input tokens include tokens corresponding to the input text.

3. The method according to any one of claims 1-2, further comprising: Increase the frequency of the corresponding token in the token knowledge base based on the first token; Reduce the frequency of the corresponding token in the token knowledge base based on the second token; The second token is the token that failed verification among the recalled tokens.

4. The method according to any one of claims 1-3, characterized in that, The recall token includes N recall drafts, each recall draft includes at least one token, and the method further includes: If N is greater than a set value, M recall drafts are obtained from the N recall drafts based on the frequency of the N recall drafts in the token knowledge base; where N is a positive integer, M is a positive integer, and N is greater than M. Accordingly, the step of using the LLM to verify the recall token and obtain a first token that passes verification includes: The first token is obtained by verifying the M recall drafts using the LLM.

5. The method according to any one of claims 1-4, characterized in that, The step of using the LLM to verify the recall token and obtain a first token that passes verification includes: The recalled tokens are merged to obtain a first draft to be verified; the length of the first draft to be verified does not exceed the length of the longest draft. The first draft to be verified is verified using the LLM and the generated token to obtain the first token.

6. The method according to claim 5, characterized in that, The method further includes: The longest draft length is adjusted based on the number of tokens in the first token.

7. The method according to claim 6, characterized in that, The step of adjusting the longest draft length based on the number of tokens in the first token includes: The receiving length is obtained based on the number of tokens in the first token and the number of tokens in the third token; the third token is a token that has passed K verifications before obtaining the first token, where K is a positive integer. If the accepted length is greater than the set length, the longest draft length is adjusted using the set length. If the accepted length is less than or equal to the set length, the longest draft length is adjusted using the accepted length.

8. The method according to claim 5, characterized in that, The first token includes a first sub-token and a second sub-token, the first draft to be verified includes the first sub-token and the second sub-token, and updating the generated token using the first token includes: If there is at least one token between the first sub-token and the second sub-token in the first draft to be verified, and the first sub-token is located before the second sub-token, the generated token is updated using the first sub-token.

9. The method according to claim 8, characterized in that, After updating the generated token using the first token, the method further includes: Use the second sub-token to generate a second draft to be verified.

10. The method according to any one of claims 1-9, characterized in that, Before using the LLM to perform inference on the input text, the method further includes: The input text is segmented to obtain the fourth token; The token knowledge base is updated using the fourth token.

11. A knowledge-based decoding device, characterized in that, The device includes: The matching module is used to match the generated identifier token of the large language model LLM with the token in the token knowledge base to obtain the recall token; the token knowledge base includes: domain knowledge tokens, which are obtained by segmenting the rewritten text, and the rewritten text is obtained by rewriting the domain knowledge text by the LLM; the generated tokens are obtained by the LLM inference from the input text. The verification module is used to verify the recall token using the LLM to obtain a first token that has passed verification; the first token is a part of the recall token; An update module is used to update the generated token using the first token.

12. The apparatus according to claim 11, characterized in that, The token knowledge base also includes: output tokens and input tokens, wherein the output tokens include tokens generated and / or verified during the inference process of the LLM, and the input tokens include tokens corresponding to the input text.

13. The apparatus according to any one of claims 11-12, further comprising: An additional module is added to increase the frequency of the corresponding token in the token knowledge base based on the first token. The reduction module is used to reduce the frequency of the corresponding token in the token knowledge base based on the second token; the second token is the token that failed verification in the recall token.

14. The apparatus according to any one of claims 11-13, characterized in that, The recall token includes N recall drafts, each recall draft includes at least one token, and the device further includes: The acquisition module is used to, if N is greater than a set value, acquire M recall drafts from the N recall drafts based on the frequency of the N recall drafts in the token knowledge base; where N is a positive integer, M is a positive integer, and N is greater than M; The verification module is specifically used to verify the M recall drafts using the LLM to obtain the first token.

15. The apparatus according to any one of claims 11-14, characterized in that, The verification module includes: A merging unit is used to merge the recalled tokens to obtain a first draft to be verified; the length of the first draft to be verified does not exceed the length of the longest draft. The verification unit is used to verify the first draft to be verified using the LLM and the generated token to obtain the first token.

16. The apparatus according to claim 15, characterized in that, The device further includes: An adjustment module is used to adjust the length of the longest draft based on the number of tokens in the first token.

17. The apparatus according to claim 16, characterized in that, The adjustment module includes: The acquisition unit is used to obtain the acceptance length based on the number of tokens in the first token and the number of tokens in the third token; the third token is a token that has passed K verifications before obtaining the first token, where K is a positive integer; An adjustment unit is used to adjust the longest draft length using the set length if the accepted length is greater than the set length. The adjustment unit is further configured to adjust the longest draft length using the accepted length if the accepted length is less than or equal to the set length.

18. The apparatus according to claim 15, characterized in that, The first token includes a first sub-token and a second sub-token, and the first draft to be verified includes the first sub-token and the second sub-token; The update module is specifically used to update the generated token using the first sub-token if there is at least one token between the first sub-token and the second sub-token in the first draft to be verified, and the first sub-token is located before the second sub-token.

19. The apparatus according to claim 18, characterized in that, The device further includes: The generation module is used to generate a second draft to be verified using the second sub-token.

20. The apparatus according to any one of claims 11-19, characterized in that, The device further includes: The word segmentation module is used to segment the input text into words to obtain the fourth token; The update module is also used to update the token knowledge base using the fourth token.

21. A knowledge-based decoding device, comprising a processor and a memory, the memory and the processor being coupled, the processor being configured to execute the knowledge-based decoding method according to any one of claims 1 to 10.

22. A computer program product containing instructions that, when run on a computer, causes the computer to perform the knowledge-based decoding method as described in any one of claims 1 to 10.

Images