Inference method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides an inference method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of natural language processing. The method comprises the steps of obtaining a prompt, at least one SSM and an LLM; inputting the prompt into each SSM in the at least one SSM, and processing the prompt through each SSM to obtain a Token tree corresponding to the SSM; when the at least one SSM is one SSM, taking the Token tree corresponding to the SSM as a target Token tree, and when the at least one SSM is multiple SSMs, merging the Token trees corresponding to the multiple SSMs to obtain the target Token tree; and starting from a root node in the target Token tree, checking the matching condition of the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, and generating a reasoning result according to the matching result. In this way, generation of the reasoning result is accelerated.

Description

Background Art

[0002] In the field of natural language processing technology, generative large language models (LLMs), such as GPT-4, have performed well in many application areas such as question answering, program synthesis, and task automation. However, LLMs have many parameters and a complex architecture, and it takes several seconds to process a single inference request, which is slow.

[0003] In the related technology, a small speculation model (SSM) is used to process prompts, output a Token (the smallest unit used to represent a text number) sequence, and use LLM to check the correctness of the Token sequence. After the check passes, the Token sequence is used as the inference result.

[0004] However, since the model capacity of SSM is much smaller than that of LLM, it is usually necessary to check multiple token sequences one by one to obtain a token sequence that can pass the check, which leads to a slower speed in obtaining inference results. Summary of the invention

[0005] The present application provides a reasoning method, device, electronic device and computer-readable storage medium, which at least to some extent overcome the problem of slow reasoning speed in related technologies.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0007] According to one aspect of the present application, there is provided an inference method for an inference device, comprising: obtaining a prompt, at least one SS and an LLM; inputting the prompt into each SSM of the at least one SSM, and processing the prompt as follows through each SSM: deriving multiple first Tokens according to the prompt; when the confidence of any first Token is greater than a first confidence threshold, continuing to derive multiple second Tokens based on any first Token; when the confidence of any second Token is greater than the first confidence threshold, continuing to derive subsequent Tokens based on any second Token until the confidence of the derived Tokens is no greater than the first confidence threshold, thereby obtaining a Token tree corresponding to the SSM; when the at least one SSM is one SSM, taking the Token tree corresponding to the SSM as the target Token tree, and when the at least one SSM is multiple SSMs, merging the Token trees corresponding to the multiple SSMs to obtain a target Token tree; starting from the root node in the target Token tree, checking the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, and generating an inference result according to the matching result.

[0008] At least one SSM is used to generate multiple first tokens at one time, and multiple second tokens and subsequent tokens are continuously generated on this basis, and organized into a target token tree. When LLM is used to verify all token sequences in the target token tree, the parameters of LLM can be traversed once to verify the consistency of all token sequences in the target token tree with the LLM output, thereby reducing the number of memory accesses to the LLM parameters and speeding up the generation of inference results.

[0009] Furthermore, by using the confidence of the Token on each node in the Token tree to determine whether to continue generating subsequent child nodes on the node, the depth of the Token tree can be dynamically adjusted, and the length of the Token sequence with a low probability of passing the verification on the target Token tree can be suppressed, thereby helping to reduce the computing resources, energy consumption, and time used in the process of generating the target Token tree and verifying the target Token tree to generate the reasoning results, further speeding up the generation of the reasoning results.

[0010] In some embodiments, in the process of checking the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, the child node of the Token with higher confidence is preferentially selected.

[0011] By giving priority to child nodes with higher confidence levels when checking the match between the token on the node and the original output, it is more likely that a token sequence matching the original output can be checked more quickly, greatly reducing the time required to verify the token tree and further speeding up the generation of inference results.

[0012] According to another aspect of the present application, another reasoning method is provided, which is applied to an reasoning device, including: obtaining a prompt, at least one SSM and an LLM; inputting the prompt into each SSM in the at least one SSM, processing the prompt through each SSM, and obtaining a Token tree corresponding to each SSM; when the at least one SSM is one SSM, using the Token tree corresponding to the SSM as the target Token tree, and when the at least one SSM is multiple SSMs, merging the Token trees corresponding to the multiple SSMs to obtain a target Token tree; starting from the root node in the target Token tree, checking the matching of the Token on the node of each Token sequence with the original output generated by the LLM according to the prompt, and generating an inference result based on the matching result, wherein when selecting a child node in the process of checking the matching, a child node with a higher confidence level of the Token is preferentially selected.

[0013] At least one SSM is used to generate multiple first tokens at one time, and multiple second tokens and subsequent tokens are continuously generated on this basis, and organized into a target token tree. When LLM is used to verify all token sequences in the target token tree, the parameters of LLM can be traversed once to verify the consistency of all token sequences in the target token tree with the LLM output, thereby reducing the number of memory accesses to the LLM parameters and speeding up the generation of inference results.

[0014] Furthermore, by giving priority to child nodes with higher confidence levels when checking the match between the Token on the node and the original output, it is more likely that a Token sequence matching the original output can be checked more quickly, greatly reducing the time required to verify the Token tree and further speeding up the generation of inference results.

[0015] In some embodiments, in the process of checking the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, when the confidence of the Token on any node is not greater than the second confidence threshold, the checking of the Token sequence where the any node is located is stopped.

[0016] By stopping the checking of the Token sequence at each node in the Token tree when the confidence of the Token at the node is not greater than the second confidence threshold, the computing resources, energy consumption, and time used in the process of verifying the target Token tree to generate reasoning results can be reduced, thereby further speeding up the generation of reasoning results.

[0017] According to another aspect of the present application, there is also provided an inference device, which is applied to an inference device, including:

[0018] The first acquisition module is used to acquire a prompt, at least one SSM and an LLM; the first processing module is used to input the prompt into each SSM of the at least one SSM, and process the prompt as follows through each SSM: derive multiple first tokens according to the prompt; when the confidence of any first token is greater than a first confidence threshold, continue to derive multiple second tokens based on the first token; when the confidence of any second token is greater than the first confidence threshold, continue to derive subsequent tokens based on the second token until the derived tokens are The confidence of oken is not greater than the first confidence threshold, and the Token tree corresponding to the SSM is obtained; the first definition and merging module is used to take the Token tree corresponding to the SSM as the target Token tree when the at least one SSM is one SSM, and merge the Token trees corresponding to the multiple SSMs to obtain the target Token tree when the at least one SSM is multiple SSMs; the first checking and generating module is used to start from the root node in the target Token tree, check the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, and generate an inference result based on the matching result.

[0019] In some embodiments, the first checking and generating module is further used to give priority to sub-nodes with higher confidence levels of Tokens when checking the matching between the Tokens on the nodes of each Token sequence and the original output generated by the LLM according to the prompt.

[0020] According to another aspect of the present application, there is also provided an inference apparatus for inference equipment, including: a second acquisition module, used to acquire a prompt, at least one SSM and an LLM; a second processing module, used to input the prompt into each SSM in the at least one SSM, process the prompt through each SSM, and obtain a Token tree corresponding to each SSM; a second definition and merging module, used to use the Token tree corresponding to the SSM as the target Token tree when the at least one SSM is one SSM, and to merge the Token trees corresponding to the multiple SSMs to obtain the target Token tree when the at least one SSM is multiple SSMs; a second checking and generating module, used to check, starting from the root node in the target Token tree, the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, and generate an inference result according to the matching result, wherein when selecting a child node in the process of checking the matching, a child node with a higher confidence level of the Token is preferentially selected.

[0021] In some embodiments, the second checking and generating module is further used to stop checking the Token sequence of any node when the confidence of the Token on any node is not greater than the second confidence threshold during the process of checking the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt.

[0022] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned reasoning methods by executing the executable instructions.

[0023] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements any one of the above-mentioned reasoning methods.

[0024] According to another aspect of the present application, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned reasoning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A schematic diagram of an inference system in one embodiment of the present application is shown.

[0027] Figure 2 A flow chart of the reasoning method in one embodiment of the present application is shown.

[0028] Figure 3 A schematic diagram showing the derivation of a Token based on confidence in one embodiment of the present application.

[0029] Figure 4 A schematic diagram of a Token tree according to an embodiment of the present application is shown.

[0030] Figure 5 A schematic diagram showing how two Token trees are merged into one Token tree in one embodiment of the present application.

[0031] Figure 6 A flow chart of the reasoning method in another embodiment of the present application is shown.

[0032] Figure 7 A schematic diagram of an inference device in one embodiment of the present application is shown.

[0033] Figure 8 A schematic diagram of an inference device in another embodiment of the present application is shown.

[0034] Fig. 9 A structural block diagram of an electronic device in one embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0036] In addition, the accompanying drawings are only schematic illustrations of the present application and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0037] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations.

[0038] To facilitate understanding, before introducing the embodiments of the present application, several terms involved in the embodiments of the present application are first explained as follows.

[0039] Token: The smallest unit used to represent the number of text. For example, in "a primary school has six grades", "one", "primary school", "there are", "six", and "grades" are all one token.

[0040] prompt: Token sequence, used to input into the model; for example, “a / primary school / has / six / grades”.

[0041] The specific implementation of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0042] Figure 1 FIG. 2 shows a schematic diagram of the reasoning system in an embodiment of the present application. Figure 1 As shown, the system may include a plurality of request devices 11 and reasoning devices 12 .

[0043] The requesting device 11 may generate and send an inference request to the inference device 12 , wherein the inference request carries a prompt.

[0044] After receiving the inference request, the inference device 12 can use one or more SSMs to infer one or more Token trees according to the prompt carried in the inference request, and obtain the target Token tree based on the one or more Token trees.

[0045] Afterwards, the inference device 12 can verify the target Token tree (ie, check the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt), and generate the inference result according to the verification (ie, matching result) result.

[0046] For information on how to infer how the device 12 implements verification of the target Token tree, please refer to the following Figure 2 The corresponding embodiments are not described in detail here.

[0047] The network is a medium used to provide a communication link between the requesting device 11 and the reasoning device 12, and can be a wired network or a wireless network.

[0048] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocols. The network is typically the Internet, but may also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec), etc. may be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies may also be used to replace or supplement the above data communication technologies.

[0049] The requesting device 11 may be various electronic devices, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, etc.

[0050] The reasoning device 12 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the request device 11. The background management server may analyze and process the received request data, and feed back the processing results to the terminal device.

[0051] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0052] Those skilled in the art will know that Figure 1 The number of requesting devices in the example is only illustrative, and any number of requesting devices may be provided according to actual needs. This embodiment of the present application does not limit this.

[0053] Under the above system architecture, an inference method is provided in an embodiment of the present application, and the method can be executed by any electronic device with computing and processing capabilities. For example, the device is an inference device.

[0054] Figure 2 A flow chart of a request device method in an embodiment of the present application is shown as follows: Figure 2 As shown, the request device method provided in the embodiment of the present application includes the following S201 to S204.

[0055] S201, obtain prompt, at least one SSM and LLM.

[0056] Among them, prompt is a hint, which can be carried by the inference request; or the inference device obtains the prompt from other devices through the network; or the inference device is equipped with an input device, through which the prompt can be input into the inference device.

[0057] The embodiments of the present application do not limit the specific content of the prompt. For example, a prompt may be "Where is the creativity of this patent application?"

[0058] The embodiment of the present application does not limit whether the at least one SSM is one SSM or multiple SSMs. The embodiment of the present application does not limit which model the SSM is, and it can be any lightweight language model.

[0059] As for how to obtain at least one SSM, the embodiments of the present application do not limit it. For example, at least one SSM can be pre-stored in the inference device, or at least one SSM can be obtained from other devices through the network, or at least one SSM can be directly constructed and trained in the inference device.

[0060] The embodiments of the present application do not limit which model the LLM is. For example, the LLM may be GPT-4, Wenxinyiyan, etc.

[0061] The specific implementation method of obtaining the LLM is not limited in the embodiments of the present application. For example, the LLM can be pre-stored in the inference device, or obtained from other devices through the network, or directly constructed and trained in the inference device to obtain the LLM.

[0062] S202, input prompt into each SSM of at least one SSM, and perform the processing shown in the following S2021-S2023 on prompt through each SSM.

[0063] S2021, derive multiple first tokens based on prompt.

[0064] The inference device inputs the prompt into the SSM, and the SSM derives multiple first tokens.

[0065] S2022: When the confidence of any first Token is greater than a first confidence threshold, continue to derive multiple second Tokens based on the first Token.

[0066] When the SSM model derives a Token based on the prompt, it can also output the confidence of the Token (SSM integrates confidence judgment) to determine whether the confidence of each first Token is greater than the first confidence threshold. When the confidence of any (any one) first Token is greater than the first confidence threshold, the second Token will continue to be derived based on the first Token.

[0067] like Figure 3 As shown, taking 5 first tokens as an example, the confidence levels are 0.7, 0.82, 0.55, 0.6 and 0.41 respectively. If the first confidence threshold is 0.6, the first tokens with confidence levels of 0.7 and 0.82 can continue to be processed by the SSM to derive second tokens. The embodiments of the present application do not impose any restriction on how many second tokens can be derived from each first token. The number of second tokens derived from different first tokens by the SSM can be the same or different, and the embodiments of the present application do not impose any restriction on this.

[0068] S2023: When the confidence of any second Token is greater than the first confidence threshold, continue to derive subsequent Tokens based on any second Token until the confidence of the derived Tokens is no greater than the first confidence threshold, thereby obtaining the Token tree corresponding to the SSM.

[0069] For example, the Token tree can be Figure 4 As shown, in Figure 4 In the Token tree shown, for each node without child nodes, the confidence of its Token is not greater than the first confidence threshold.

[0070] The embodiments of the present application do not limit the specific value of the first confidence threshold, and it can be set based on experience.

[0071] S203, when at least one SSM is one SSM, the Token tree corresponding to the SSM is used as the target Token tree; when at least one SSM is multiple SSMs, the Token trees corresponding to the multiple SSMs are merged to obtain the target Token tree.

[0072] When at least one SSM is multiple SSMs, the embodiments of the present application do not limit how to merge the Token trees corresponding to the multiple SSMs. For example, multiple Token trees can be merged by voting, stacking, etc.

[0073] To merge two Token trees into one Token tree, see Figure 5 Among them, A is the content of the Token in the root node, and B1-B3 and C1-C4 are all the contents of the Token.

[0074] S204, starting from the root node in the target Token tree, check the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, and generate an inference result according to the matching result.

[0075] In one embodiment, the method used to check the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt can be greedy decoding or random decoding, which is not limited in the embodiments of the present application.

[0076] Taking the greedy decoding method to check the matching of the token on the node of each token sequence with the original output generated by LLM according to the prompt as an example, starting from the root node of 𝒩 (the target token tree), iteratively check whether the corresponding token of the node is consistent with the original output of LLM. For a node u∈𝒩, if u contains a child node v, and the token corresponding to v is consistent with the output of LLM (that is, t v = O(u), t vis the Token corresponding to node v, O(u) is the next Token generated by LLM based on the Token of node u (that is, the Token at the child node of u). If the token is matched, the verification of node u is completed, and the child node v is checked. When node u does not contain a child node of the LLM output result (the output Token), O(u) is added as a verified node in 𝒩, and the verification process is terminated. Finally, all verified nodes are appended to the currently generated Token sequence 𝒱 to obtain 𝒱, and the Token sequence in 𝒱 is output as the inference result.

[0077] At least one SSM is used to generate multiple first tokens at one time, and multiple second tokens and subsequent tokens are continuously generated on this basis, and organized into a target token tree. When LLM is used to verify all token sequences in the target token tree, the parameters of LLM can be traversed once to verify the consistency of all token sequences in the target token tree with the LLM output, thereby reducing the number of memory accesses to the LLM parameters and speeding up the generation of inference results.

[0078] Furthermore, by using the confidence of the token on each node in the Token tree to determine whether to continue generating subsequent child nodes on the node, the depth of the Token tree can be dynamically adjusted, reducing the token sequence with a low probability of passing the verification on the target Token tree, thereby helping to reduce the computing resources, energy consumption, and time used in the process of generating the target Token tree and verifying the target Token tree to generate the reasoning results, further speeding up the generation of reasoning results.

[0079] In one embodiment, in the process of checking the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, the child node of the Token with higher confidence is preferentially selected.

[0080] For example, when checking the matching situation, if the number of first tokens is 5, then the first token with the highest confidence level is preferentially selected from the 5 first tokens for detection.

[0081] For example, Figure 5As shown, the Token tree includes three layers. The first layer is the root node, the second layer is two first Tokens with confidences of 0.5 and 0.7 respectively, and the third layer is three second Tokens with confidences of 0.7, 0.4 and 0.9 respectively. When checking the matching situation, the first Token with a confidence of 0.7 is first selected for detection. If it matches, the second Token with a confidence of 0.9 is selected for detection. If it does not match, the second Token with a confidence of 0.7 is selected for matching. If it matches, a Token sequence consisting of a first Token with a confidence of 0.7 and a second Token with a confidence of 0.7 is obtained, and the Token sequence is used as the inference result.

[0082] By giving priority to child nodes with higher confidence levels when checking the match between the token on the node and the original output, it is more likely that a token sequence matching the original output can be checked more quickly, greatly reducing the time required to verify the token tree and further speeding up the generation of inference results.

[0083] Another inference method is also provided in the embodiment of the present application, and the method can be executed by any electronic device with computing and processing capabilities. For example, the device is an inference device.

[0084] Figure 6 A flow chart of a request device method in an embodiment of the present application is shown as follows: Figure 6 As shown, the request device method provided in the embodiment of the present application includes the following S601 to S604.

[0085] S601, obtain prompt, at least one SSM and LLM.

[0086] S602, input the prompt into each SSM of at least one SSM, process the prompt through each SSM, and obtain a Token tree corresponding to each SSM.

[0087] By extending the method, when SSM is used to process prompts, multiple possible tokens are output at each step. These multiple possible tokens can limit the structure of the token tree by limiting the number of tokens generated at each step. For example, if the number of decryption steps is m, then<k1、k2、……、ki……、km> Here, ki represents the number of tokens expanded by each token in the i-th step.

[0088] S603, when at least one SSM is one SSM, the Token tree corresponding to the SSM is used as the target Token tree; when at least one SSM is multiple SSMs, the Token trees corresponding to the multiple SSMs are merged to obtain the target Token tree.

[0089] S604, starting from the root node in the target Token tree, check the matching between the Token on the node of each Token sequence and the original output generated by LLM according to the prompt, and generate an inference result based on the matching result. When selecting child nodes in the process of checking the matching situation, child nodes with higher confidence levels of the Token are given priority.

[0090] At least one SSM is used to generate multiple first tokens at one time, and multiple second tokens and subsequent tokens are continuously generated on this basis, and organized into a target token tree. When LLM is used to verify all token sequences in the target token tree, the parameters of LLM can be traversed once to verify the consistency of all token sequences in the target token tree with the LLM output, thereby reducing the number of memory accesses to the LLM parameters and speeding up the generation of inference results.

[0091] Furthermore, by giving priority to child nodes with higher confidence levels when checking the match between the Token on the node and the original output, it is more likely that a Token sequence matching the original output can be checked more quickly, greatly reducing the time required to verify the Token tree and further speeding up the generation of inference results.

[0092] In one embodiment, in the process of checking the matching between the Token on the node of each Token sequence and the original output generated by LLM according to the prompt, if the confidence of the Token on any node is not greater than the second confidence threshold, the checking of the Token sequence where the any node is located is stopped.

[0093] By stopping the checking of the Token sequence at each node in the Token tree when the confidence of the Token at the node is not greater than the second confidence threshold, the computing resources, energy consumption, and time used in the process of verifying the target Token tree to generate reasoning results can be reduced, thereby further speeding up the generation of reasoning results.

[0094] The embodiment of the present application does not limit the specific value of the second confidence threshold, and it can be set according to experience. The second confidence threshold is equal to the first confidence threshold.

[0095] Based on the same inventive concept, the present application also provides two inference devices in the embodiments, such as the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0096] Figure 7 A schematic diagram of an inference device in an embodiment of the present application is shown. Figure 7 As shown, the device is applied to an inference device, including: a first acquisition module 71, used to acquire a prompt, at least one SSM and an LLM; a first processing module 72, used to input the prompt into each SSM in at least one SSM, and process the prompt as follows through each SSM: derive multiple first Tokens according to the prompt; when the confidence of any first Token is greater than a first confidence threshold, continue to derive multiple second Tokens based on any first Token; when the confidence of any second Token is greater than the first confidence threshold, continue to derive subsequent Tokens based on any second Token. n, until the confidence of the derived Token is no more than the first confidence threshold, and the Token tree corresponding to the SSM is obtained; the first definition and merging module 73 is used to use the Token tree corresponding to the SSM as the target Token tree when at least one SSM is one SSM, and to merge the Token trees corresponding to the multiple SSMs to obtain the target Token tree when at least one SSM is multiple SSMs; the first checking and generating module 74 is used to start from the root node in the target Token tree, check the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, and generate an inference result based on the matching result.

[0097] In some embodiments, the first checking and generating module 74 is further used to give priority to selecting child nodes with higher confidence levels of the Token when checking the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt.

[0098] At least one SSM is used to generate multiple first tokens at one time, and multiple second tokens and subsequent tokens are continuously generated on this basis, and organized into a target token tree. When LLM is used to verify all token sequences in the target token tree, the parameters of LLM can be traversed once to verify the consistency of all token sequences in the target token tree with the LLM output, thereby reducing the number of memory accesses to the LLM parameters and speeding up the generation of inference results.

[0099] Furthermore, by using the confidence of the token on each node in the Token tree to determine whether to continue generating subsequent child nodes on the node, the depth of the Token tree can be dynamically adjusted, reducing the token sequence with a low probability of passing the verification on the target Token tree, thereby helping to reduce the computing resources, energy consumption, and time used in the process of generating the target Token tree and verifying the target Token tree to generate the reasoning results, further speeding up the generation of reasoning results.

[0100] Figure 8 Another schematic diagram of an inference device in an embodiment of the present application is shown as follows: Figure 8 As shown, the device is applied to an inference device, including:

[0101] The second acquisition module 81 is used to obtain prompt, at least one SSM and LLM; the second processing module 82 is used to input prompt into each SSM in at least one SSM, process prompt through each SSM, and obtain the Token tree corresponding to each SSM; the second definition and merging module 83 is used to use the Token tree corresponding to the SSM as the target Token tree when at least one SSM is one SSM, and merge the Token trees corresponding to multiple SSMs to obtain the target Token tree when at least one SSM is multiple SSMs; the second checking and generating module 84 is used to start from the root node in the target Token tree, check the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, and generate an inference result according to the matching result, wherein when selecting a child node in the process of checking the matching, the child node with a higher confidence level of the Token is given priority.

[0102] In some embodiments, the second checking and generating module 84 is also used to stop checking the Token sequence of any node when the confidence of the Token on any node is not greater than the second confidence threshold during the process of checking the matching between the Token on the node of each Token sequence and the original output generated by LLM according to the prompt.

[0103] At least one SSM is used to generate multiple first tokens at one time, and multiple second tokens and subsequent tokens are continuously generated on this basis, and organized into a target token tree. When LLM is used to verify all token sequences in the target token tree, the parameters of LLM can be traversed once to verify the consistency of all token sequences in the target token tree with the LLM output, thereby reducing the number of memory accesses to the LLM parameters and speeding up the generation of inference results.

[0104] Furthermore, by giving priority to child nodes with higher confidence levels when checking the match between the Token on the node and the original output, it is more likely that a Token sequence matching the original output can be checked more quickly, greatly reducing the time required to verify the Token tree and further speeding up the generation of inference results.

[0105] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0106] Refer to the following Fig. 9 hereinafter describes an electronic device 900 according to this embodiment of the present application. Fig. 9 The electronic device 900 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0107] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).

[0108] The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps of various exemplary embodiments of the present application described in the above "Exemplary Method" section of this specification. For example, the processing unit 910 can perform the following steps of the above method embodiment: S201~S204, S601~S604.

[0109] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0110] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0111] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0112] The electronic device 900 may also communicate with one or more external devices 940 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication may be performed through an input / output (I / O) interface 950. In addition, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 through a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0113] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0114] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart may be implemented as a computer program product, which includes: a computer program, which implements the above reasoning method when executed by a processor.

[0115] In an exemplary embodiment of the present application, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. The computer-readable storage medium stores a program product capable of implementing the above method of the present application.

[0116] In some possible implementations, various aspects of the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present application described in the above “Exemplary Method” section of this specification.

[0117] More specific examples of computer-readable storage media in the present application may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0118] In the present application, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0119] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0120] In specific implementation, the program code for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on the remote computing device, or completely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0121] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0122] In addition, although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0123] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0124] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the appended claims.

Claims

1. A reasoning method, characterized in that: Applied to inference devices, including: Get a prompt, at least a small speculation model SSM and a large language model LLM; The prompt is input into each SSM of the at least one SSM, and the prompt is processed by each SSM as follows: Deriving multiple first Tokens according to the prompt; When the confidence of any first Token is greater than a first confidence threshold, continue to derive multiple second Tokens based on the first Token; When the confidence of any second Token is greater than the first confidence threshold, continue to derive subsequent Tokens based on the second Token until the confidence of the derived Tokens is no greater than the first confidence threshold, thereby obtaining a Token tree corresponding to the SSM; When the at least one SSM is one SSM, the Token tree corresponding to the SSM is used as the target Token tree; when the at least one SSM is multiple SSMs, the Token trees corresponding to the multiple SSMs are merged to obtain the target Token tree; Starting from the root node in the target Token tree, check the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, and generate an inference result according to the matching result.

2. The inference method according to claim 1, characterized in that: In the process of checking the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, the child node with a higher confidence of the Token is preferentially selected.

3. A reasoning method, characterized in that: Applied to inference devices, including: Get a prompt, at least a small speculation model SSM and a large language model LLM; Input the prompt into each SSM of the at least one SSM, and process the prompt through each SSM to obtain a Token tree corresponding to each SSM; When the at least one SSM is one SSM, the Token tree corresponding to the SSM is used as the target Token tree; when the at least one SSM is multiple SSMs, the Token trees corresponding to the multiple SSMs are merged to obtain the target Token tree; Starting from the root node in the target Token tree, check the matching of the Token on the node of each Token sequence with the original output generated by the LLM according to the prompt, and generate an inference result based on the matching result, wherein when selecting child nodes in the process of checking the matching situation, give priority to selecting child nodes with higher confidence levels of the Token.

4. The inference method according to claim 3, characterized in that: In the process of checking the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, when the confidence of the Token on any node is not greater than the second confidence threshold, stop checking the Token sequence where the any node is located.

5. An inference device, characterized in that: Applied to inference devices, including: A first acquisition module is used to acquire a prompt, at least one small speculation model SSM and a large language model LLM; A first processing module is used to input the prompt into each SSM of the at least one SSM, and process the prompt as follows through each SSM: derive multiple first Tokens according to the prompt; when the confidence of any first Token is greater than a first confidence threshold, continue to derive multiple second Tokens based on any first Token; when the confidence of any second Token is greater than the first confidence threshold, continue to derive subsequent Tokens based on any second Token, until the confidence of the derived Tokens is not greater than the first confidence threshold, and obtain a Token tree corresponding to the SSM; A first definition and merging module is used to, when the at least one SSM is one SSM, use the Token tree corresponding to the SSM as the target Token tree, and when the at least one SSM is multiple SSMs, merge the Token trees corresponding to the multiple SSMs to obtain the target Token tree; The first checking and generating module is used to check the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, starting from the root node in the target Token tree, and generate an inference result according to the matching result.

6. The inference device according to claim 5, characterized in that The checking and generating module is further used to preferentially select child nodes with higher confidence levels of Tokens in the process of checking the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt.

7. An inference device, characterized in that: Applied to inference devices, including: The second acquisition module is used to acquire a prompt, at least one small speculation model SSM and a large language model LLM; A second processing module is used to input the prompt into each SSM of the at least one SSM, and process the prompt through each SSM to obtain a Token tree corresponding to each SSM; A second definition and merging module is used to, when the at least one SSM is one SSM, use the Token tree corresponding to the SSM as the target Token tree, and when the at least one SSM is multiple SSMs, merge the Token trees corresponding to the multiple SSMs to obtain the target Token tree; The second checking and generating module is used to check the matching between the Token on the node of each Token sequence and the original output generated by the LLM according to the prompt, starting from the root node in the target Token tree, and generate an inference result based on the matching result, wherein when selecting a child node in the process of checking the matching situation, a child node with a higher confidence level of the Token is preferentially selected.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the reasoning method according to any one of claims 1 to 4 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the inference method according to any one of claims 1 to 4 is implemented.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the reasoning method according to any one of claims 1 to 4.

Images