Large model reasoning method and system based on adjacency matrix and speculative decoding

Abstract

The invention discloses a large model reasoning method and system based on an adjacent matrix and speculative decoding, and belongs to the technical field of large model reasoning. The problems that the acceleration capacity of an existing method is limited, and other candidate words generated in the decoding process of a large model are not fully utilized when a draft word sequence is constructed are solved. The method specifically comprises the following steps of: 1, constructing an adjacent matrix for storing k candidate words of each word in a word list; 2, retrieving a draft tree composed of draft word sequences from the adjacent matrix according to the input query sequence; 3, constructing a compressed sequence by using nodes in the draft tree, and constructing an attention mask matrix based on the draft tree; and step 4, inputting the attention mask matrix and the compressed sequence into a large model, verifying each draft word sequence in the draft tree by using the large model to obtain a reasoning result, and updating the adjacent matrix. The method can be applied to large model reasoning.

Description

Technical Field

[0001] The present invention belongs to the technical field of large model reasoning, and in particular relates to a large model reasoning method and system based on adjacency matrix and speculative decoding. Background Art

[0002] The rapid growth of large model parameters has led to an increasing inference delay, which limits the application of large models in real-world scenarios. Currently, all large models are decoder architectures, and their default decoding strategy is autoregressive decoding. Given an input sequence s = (x0, ..., x t ), the model calculates The probability distribution p on t+1 =P(x|s;θ), where θ represents the large model parameters. The model is then t+1 Select the word x with the highest probability t+1 This decoding strategy causes the model to calculate only one word at a time, limiting the model's reasoning efficiency.

[0003] like Figure 1 As shown in Figure 1, speculative decoding makes full use of the parallel computing capabilities of modern hardware devices by guessing first and then verifying, which can accelerate the reasoning process of large models without loss. Speculative decoding first guesses some draft words based on the input sequence s And put the guessed draft word after the input sequence s, and get pt in one calculation +1 and Due to the parallel computing capability of the hardware, under a certain number of draft words, the computation time is basically the same as that of autoregressive decoding. Based on these probability distributions, we can get multiple words x t+1 and Let's verify the correctness of the draft word. Specifically, if We consider draft words Successfully hit the correct answer and continue to verify Since the last draft word was successfully hit, it can be considered and p in the autoregressive decoding process t+2 Equal, so you can continue to verify and x t+2 The subsequent verification process is similar, so that we can get a word sequence with The continuous correct segments at the beginning make it possible to obtain multiple correct words in one decoding process, thereby greatly accelerating the reasoning process of large models.

[0004] The draft word acquisition methods of speculative decoding can be divided into training methods and retrieval methods. The training methods need to add additional components to the model for training, such as additional small models or parameter-efficient architectures. Therefore, they cannot be plug-and-play and the components between different models are not universal and need to be adapted separately. The retrieval method does not require training components and is relatively universal. The existing main retrieval methods include:

[0005] 1. Prompt Lookup-Decoding (PLD): Check whether the suffix of the current output exists in the previous text. If so, use the content after the "suffix" in the previous text as the draft word sequence.

[0006] 2. Lookahead: Let the model generate W N-grams based on W N-1grams each time it decodes. When the specified length is reached, the Ngram is put into the search pool, and the N-1gram of the first word is removed for the next generation of Ngram. Each time, the search pool is searched for a word sequence whose first word is the same as the last word of the input, and the retrieved word sequence is used as the draft word sequence.

[0007] 3. REST: Use a large number of existing corpora to construct a retrieval library, and each time retrieve the most frequently occurring word sequence in the retrieval library according to the current input suffix as the draft word sequence.

[0008] Although the PLD method is low-cost, it can only retrieve from the previous text, which makes the retrieval space relatively limited, and it cannot predict new words or new word combinations. In addition, when the generated content overlaps less with the previous text, the acceleration ability is limited; the Lookahead method must have the participation of a large model when generating N-grams, which undoubtedly reduces the reasoning efficiency of the large model, and the quality of N-gram generation is not high, resulting in a low hit rate; the retrieval library used by the REST method is static and not flexible when used, and is limited by the relevance of the retrieval library to the current input and the output habits of the current model. If the content of the retrieval library is not very relevant to the current input topic, it is difficult to retrieve useful draft word sequences, or the retrieved content is not aligned with the model, resulting in these drafts not being adopted by the model in the verification phase. In addition, the scale of the retrieval library is usually large, which makes its retrieval time relatively long, and the overall acceleration benefits are very low.

[0009] In summary, the acceleration capability of existing methods is still relatively limited, and existing methods only select the word with the highest probability as the answer, and do not fully utilize other candidate words generated by the large model during the decoding process when constructing the draft word sequence. Summary of the invention

[0010] The purpose of the present invention is to solve the problem that the acceleration capability of the existing method is limited and other candidate words generated by the large model in the decoding process are not fully utilized when constructing the draft word sequence, and a large model reasoning method and system based on adjacency matrix and speculative decoding are proposed.

[0011] The technical solution adopted by the present invention to solve the above technical problems is:

[0012] According to one aspect of the present invention, a large model reasoning method based on adjacency matrix and speculative decoding, the method specifically comprises the following steps:

[0013] Step 1: Select the first k candidate words for each word in the vocabulary, and construct an adjacency matrix M for storing the k candidate words of each word. The dimension of M is in, is the number of words in the vocabulary;

[0014] Step 2: According to the last word in the input query sequence, a draft tree consisting of a draft word sequence is retrieved from the adjacency matrix M;

[0015] Step 3: Use the nodes in the draft tree to construct a compressed sequence, and construct an attention mask matrix based on the draft tree;

[0016] Step 4: Input the attention mask matrix and the compressed sequence into the large model, use the output of the large model to verify each draft word sequence in the draft tree, and update the adjacency matrix;

[0017] After entering a new query sequence, return to step 2.

[0018] According to another aspect of the present invention, a large model reasoning system based on adjacency matrix and speculative decoding, the system includes a candidate word storage module, a draft tree generation module, an attention mask matrix generation module and a large model reasoning module;

[0019] The candidate word storage module is used to store the first k candidate words of each word in the vocabulary in the adjacency matrix M;

[0020] The draft tree generation module is used to retrieve a draft tree consisting of a draft word sequence from the adjacency matrix M according to the last word in the input query sequence;

[0021] The draft word sequence retrieved from the adjacency matrix M is specifically:

[0022] The last word in the input query sequence is recorded as x0, and any draft word sequence retrieved is recorded as (x0, x1, x2, ..., x i-1 ), i is the number of levels of the draft tree;

[0023] The draft word sequence (x0, x1, x2, ..., x i-1 ) satisfies: In the adjacency matrix M, x1 is any one of the first k candidate words of x0, x2 is any one of the first k1 candidate words of x1, and x i-1 is x i-2 The first k i-2 Any one of the candidate words, and k≥k1…≥k i-2 ;

[0024] The attention mask matrix generation module is used to construct a compressed sequence using nodes in the draft tree, and construct an attention mask matrix based on the draft tree;

[0025] The large model reasoning module is used to verify each draft word sequence in the draft tree according to the input attention mask matrix and compression sequence, and update the adjacency matrix.

[0026] The beneficial effects of the present invention are:

[0027] The method of the present invention uses an adjacency matrix to store multiple candidate words for each word in a vocabulary, and retrieves a draft tree on the adjacency matrix, then uses a tree attention mechanism to verify the draft word sequence in the draft tree, and uses new candidate words generated during decoding to update the adjacency matrix, and does not require additional model structure or training, so it is highly adaptable and can be used for models of any architecture or any size, and can be seamlessly integrated into a large model of any structure or any number of parameters. During the retrieval process, since the candidate words are continuously updated with the decoding process of the large model, the entire retrieval space is dynamic, so other candidate words generated by the large model during the decoding process can be fully utilized, so that the method of the present invention has stronger flexibility than the prior art.

[0028] The method of the present invention accelerates the reasoning process by making full use of the candidate words in the decoding process, and can achieve about 2 times the reasoning acceleration, which is more than 31% higher than other methods that do not require training. It is better than other methods in terms of reasoning acceleration, and only consumes less than 2MB of additional space, which significantly reduces the demand for space resources compared to existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the speculative decoding process;

[0030] In the figure, draft token is a draft word, accepted token is a draft word that is verified to be correct, and rejected token is a draft word that is verified to be incorrect;

[0031] Figure 2is a schematic diagram of tree-like attention;

[0032] In the figure, Draft Tree represents the draft tree, Tree Attention Mask represents the tree-like attention mask matrix, and Token Sequence Verify represents the word sequence verification;

[0033] Figure 3 is an exemplary diagram of the method of the present invention. DETAILED DESCRIPTION

[0034] Specific implementation method 1: This implementation method describes a large model reasoning method based on adjacency matrix and speculative decoding, and the method specifically includes the following steps:

[0035] Step 1: Select the first k candidate words for each word in the vocabulary, and construct an adjacency matrix M for storing the k candidate words of each word. The dimension of M is in, is the number of words in the vocabulary;

[0036] Step 2: According to the last word in the input query sequence, a draft tree consisting of a draft word sequence is retrieved from the adjacency matrix M using a breadth-first search (BFS);

[0037] In the draft tree, the last word in the input query sequence is used as the root node of the tree, the candidate words of the root node retrieved from the adjacency matrix are used as the nodes of the second layer in the tree, and then the candidate words of the second layer nodes are retrieved, and the candidate words of the second layer nodes are used as the third layer nodes of the tree;

[0038] Step 3: Use the nodes in the draft tree to construct a compressed sequence, and construct an attention mask matrix based on the draft tree;

[0039] Step 4: Input the attention mask matrix and the compressed sequence into the large model, use the output of the large model to verify each draft word sequence in the draft tree, and update the adjacency matrix;

[0040] When a new query sequence is input, the updated adjacency matrix is ​​used to return to step 2.

[0041] It is observed that if the current input word appears in the subsequent generated sequence, then the subsequent word may be a candidate word generated in the previous decoding process. This shows that in addition to the word with the highest probability in each decoding process, other candidate words are also useful. The method of the present invention uses an adjacency matrix to store the candidate words output by the model for each word during the last decoding, and uses a method similar to breadth-first search (BFS) to retrieve a draft tree from the adjacency matrix. The newly generated candidate words after the model calculation can update the adjacency matrix. This method makes full use of the candidate words generated by the model in each decoding process. The retrieval space is more diverse, the words covered are more extensive, and it does not rely on the overlap of the previous text; the information used is the generated content of the model, and no additional processing and construction are required; the retrieval library is constantly updated with the decoding process, and is flexible. At the same time, the retrieval library only needs Bytes in size (int64 requires 4 bytes for storage), and due to the small space occupancy and matrix parallelization, the retrieval time is very short.

[0042] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that the adjacency matrix M is initialized according to common expressions and common patterns in various queries.

[0043] The other steps and parameters are the same as those in the first embodiment.

[0044] Initially, all elements in the adjacency matrix M are set to zero. This initialization causes the adjacency matrix to be unable to retrieve useful words at the beginning, thereby reducing the efficiency of the early reasoning stage. In order to alleviate this problem, the present invention adopts a hot start strategy, which utilizes common expressions and patterns that often appear in various queries to initialize the adjacency matrix. The hot start strategy ensures that the adjacency matrix can include more candidate words in the initial stage, thereby covering a wider range of potential draft word sequences. And the adjacency matrix is ​​constantly updated, and subsequent queries that arrive are based on the previous adjacency matrix to obtain the draft word sequence. Compared with other data storage forms, the matrix can realize efficient parallel processing of candidate words, which is crucial for reducing retrieval and update time.

[0045] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that, in step two, the draft word sequence retrieved from the adjacency matrix M is specifically:

[0046] The last word in the input query sequence is recorded as x0, and any draft word sequence retrieved is recorded as (x0, x1, x2, ..., x i-1 ), i is the number of layers of the draft tree; in the draft tree, x0 is the root node of the draft tree, x1 is at the second layer of the draft tree, x2 is at the third layer of the draft tree, ..., x i-1Located at the i-th level of the draft tree, there may be several identical draft words between any two draft word sequences;

[0047] The draft word sequence (x0, x1, x2, ..., x i-1 ) satisfies: In the adjacency matrix M, x1 is any one of the first k candidate words of x0, x2 is any one of the first k1 candidate words of x1, and x i-1 is x i-2 The first k i-2 Any one of the candidate words, and k≥k1…≥k i-2 .

[0048] The other steps and parameters are the same as those in the first or second embodiment.

[0049] The adjacency matrix M is used to store candidate words. When a word appears, its corresponding candidate word can be used as a draft word. Directly using the adjacency matrix can only determine the next word immediately adjacent. In fact, the adjacency matrix also contains subsequent candidates for these candidate words. Therefore, the sequence can be gradually extended to obtain a longer draft word sequence, and by storing the top k candidate words, multiple potential candidate paths can be explored in parallel for each word. This process similar to breadth-first search (BFS) can explore sequence length and diverse paths while using the adjacency matrix to build a draft tree. The draft tree is static and unbalanced. Static means that the number of child nodes of each node remains unchanged during all decoding processes. This fixed structure is conducive to preprocessing, such as calculating the attention mask matrix, and allows more efficient parallel operations during layer traversal; unbalanced means that the nodes in the front and previous layers in each layer will have more child nodes and deeper depths, k≥k1…≥k i-2 . Since the candidate words are sorted by their probability in the adjacency matrix, this design prioritizes the most likely paths and allocates computing resources to paths with higher confidence. The method used to retrieve the draft tree is similar to breadth-first search. The retrieval process starts from the root, which is the last word of the current input word sequence, and then expands the root node and its child nodes using the breadth-first search method, stopping after reaching the specified depth. Finally, a compressed sequence is constructed by traversing the draft tree layer by layer and sent to the large model.

[0050] Specific implementation method 4: This implementation method is different from one of the specific implementation methods 1 to 3 in that, in step 3, a compression sequence is constructed using the nodes in the draft tree. The specific process is as follows:

[0051] The root node of the draft tree is taken as the first word in the compressed sequence. In the compressed sequence, the second-level nodes of the draft tree are located after the first word, and the third-level nodes of the draft tree are located after the second-level nodes, until all nodes in the draft tree are traversed to obtain a compressed sequence.

[0052] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0053] The present invention takes the root node as the first layer of the draft tree, assumes that the second layer has two nodes, which are respectively recorded as node 1 and node 2, and the third layer has three nodes, which are respectively recorded as node 3, node 4 and node 5, and node 1 is located on the left side of node 2 in the second layer of the draft tree, node 3 is located on the left side of node 4 in the third layer of the draft tree, and node 4 is located on the left side of node 5 in the third layer of the draft tree. Then, in the compressed sequence, the order of the words of each node is the root node, node 1, node 2, node 3, node 4 and node 5.

[0054] Specific implementation method 5: This implementation method is different from any one of specific implementation methods 1 to 4 in that the attention mask matrix is ​​constructed based on the draft tree, and the attention mask matrix satisfies:

[0055] (1) All elements on the diagonal of the attention mask matrix are 1;

[0056] (2) If the lth word in the compressed sequence and the l′th word in the compressed sequence are in the same draft word sequence, and the lth word is not before the l′th word in the draft word sequence (i.e., the lth word is after the l′th word, or l=l′), then the element in the lth row and l′th column of the attention mask matrix is ​​1; l=1, 2, ..., L, l′=1, 2, ..., L, where L is the number of words in the compressed sequence;

[0057] (3) Except for the elements with values ​​1 determined in (1) and (2), all other elements in the attention mask matrix are 0.

[0058] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0059] The existing technology can only verify one sequence in one decoding process, which limits the verification efficiency. In order to verify multiple possible draft word sequences at the same time, the draft tree needs to be compressed into a sequence, such as Figure 2 As shown in , this draft tree is compressed into a sequence "for ij in, in". In the subsequent tree-like attention mask matrix, the words that each word can see are specially designed, such as Figure 2 In the attention mask matrix, the word j in the third row can only see for and itself, but not i, which means Figure 2The possible sequence on the right is "for j", while the word in in the last line only has for and j, which represents the possible sequence "fir j in". In this way, we compress multiple possible sequences into one sequence. After the model calculation, we extract all possible sequences according to the draft tree for verification and select the correct and longest sequence as the answer.

[0060] Specific implementation method 6: This implementation method is different from the specific implementation methods 1 to 5 in that the output of the large model is used to verify each draft word sequence in the draft tree and update the adjacency matrix; the specific process is:

[0061] Step 1: record the last word in the input query sequence as x0, the large model outputs the first k candidate words of word x0, and record the candidate word with the highest probability as x′1. Use the first k candidate words of word x0 output by the large model to update the candidate words of word x0 in the adjacency matrix, and continue to verify each draft word sequence whose second word in the draft tree is x′1. Each draft word sequence whose second word in the draft tree is not x′1 is an error sequence and does not need to be verified further.

[0062] Step 2, initialize a=1;

[0063] Step 3: The large model outputs word x′ a The first k candidate words of , and the candidate word with the highest probability is recorded as x′ a+1 , using the word x′ output by the large model a The first k candidate words in the adjacency matrix of word x′ a The candidate words are updated, and the second word in the draft tree is x′1, the third word is x′2, ..., the a+2th word is x′ a+1 The individual draft word sequences continue to be verified;

[0064] That is, the sequence for continuing verification should satisfy the following conditions at the same time: the second word is x′1, the third word is x′2, ..., the a+2th word is x′ a+1 ; Other draft word sequences are error sequences that do not need to be verified further;

[0065] Step 4, let a=a+1, return to step 3, until all draft word sequences are verified, and then execute step 5;

[0066] Step 5: Select the longest draft word sequence from the verified draft word sequences (the verified draft word sequence in the present invention means that each word in the sequence is the same as the word with the highest probability output by the large model in each step), and the selected draft word sequence is the inference result of the large model.

[0067] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0068] The present invention verifies the draft word sequence and selects the longest correct sequence as the final output. After the verification is completed, the top k words with the highest probability are obtained according to the probability distribution of the output draft words, so as to update the adjacency matrix. In the updating process, we directly overwrite the previous candidate words and use the new candidate words to provide draft words for the subsequent decoding steps, which enables the retrieval space to dynamically adapt to the current content, so that it focuses on the most relevant and most likely subsequent paths.

[0069] Specific implementation method seven: A large model inference system based on adjacency matrix and speculative decoding described in this implementation method includes a candidate word storage module, a draft tree generation module, an attention mask matrix generation module and a large model inference module;

[0070] The candidate word storage module is used to store the first k candidate words of each word in the vocabulary in the adjacency matrix M;

[0071] The draft tree generation module is used to retrieve a draft tree consisting of a draft word sequence from the adjacency matrix M according to the last word in the input query sequence;

[0072] The draft word sequence retrieved from the adjacency matrix M is specifically:

[0073] The last word in the input query sequence is recorded as x0, and any draft word sequence retrieved is recorded as (x0, x1, x2, ..., x i-1 ), i is the number of levels of the draft tree;

[0074] The draft word sequence (x0, x1, x2, ..., x i-1 ) satisfies: In the adjacency matrix M, x1 is any one of the first k candidate words of x0, x2 is any one of the first k1 candidate words of x1, and x i-1 is x i-2 The first k i-2 Any one of the candidate words, and k≥k1…≥k i-2 ;

[0075] The attention mask matrix generation module is used to construct a compressed sequence using nodes in the draft tree, and construct an attention mask matrix based on the draft tree;

[0076] The large model reasoning module is used to verify each draft word sequence in the draft tree according to the input attention mask matrix and compression sequence, and update the adjacency matrix.

[0077] Specific implementation eight: This implementation differs from specific implementation seven in that the adjacency matrix M is initialized according to common expressions and common patterns in various queries.

[0078] The other steps and parameters are the same as those in the seventh embodiment.

[0079] Specific implementation method 9: This implementation method is different from specific implementation methods 7 or 8 in that the working process of the attention mask matrix generation module is:

[0080] Step 1: Use the nodes in the draft tree to construct a compressed sequence;

[0081] The root node of the draft tree is taken as the first word in the compressed sequence. In the compressed sequence, the second-level nodes of the draft tree are located after the first word, and the third-level nodes of the draft tree are located after the second-level nodes, until all nodes in the draft tree are traversed to obtain a compressed sequence.

[0082] Step 2: Construct an attention mask matrix based on the draft tree. The attention mask matrix satisfies:

[0083] (1) All elements on the diagonal of the attention mask matrix are 1;

[0084] (2) If the lth word in the compressed sequence and the l′th word in the compressed sequence are in the same draft word sequence, and the lth word is not before the l′th word in the draft word sequence (i.e., the lth word is after the l′th word, or l=l′), then the element in the lth row and l′th column of the attention mask matrix is ​​1; l=1, 2, ..., L, l′=1, 2, ..., L, where L is the number of words in the compressed sequence;

[0085] (3) Except for the elements with values ​​1 determined in (1) and (2), all other elements in the attention mask matrix are 0.

[0086] The other steps and parameters are the same as those in the seventh or eighth specific implementation mode.

[0087] Specific implementation method 10: This implementation method is different from any one of specific implementation methods 7 to 9 in that the working process of the large model reasoning module is as follows:

[0088] Step 1: record the last word in the input query sequence as x0, the large model outputs the first k candidate words of word x0, and record the candidate word with the highest probability as x′1. Use the first k candidate words of word x0 output by the large model to update the candidate words of word x0 in the adjacency matrix, and continue to verify each draft word sequence whose second word in the draft tree is x′1. Each draft word sequence whose second word in the draft tree is not x′1 is an error sequence and does not need to be verified further.

[0089] Step 2, initialize a=1;

[0090] Step 3: The large model outputs word x′ a The first k candidate words of , and the candidate word with the highest probability is recorded as x′ a+1 , using the word x′ output by the large model a The first k candidate words in the adjacency matrix of word x′ a The candidate words are updated, and the second word in the draft tree is x′1, the third word is x′2, ..., the a+2th word is x′ a+1 The individual draft word sequences continue to be verified;

[0091] That is, the sequence for continuing verification should satisfy the following conditions at the same time: the second word is x′1, the third word is x′2, ..., the a+2th word is x′ a+1 ; Other draft word sequences are error sequences that do not need to be verified further;

[0092] Step 4, let a=a+1, return to step 3, until all draft word sequences are verified, and then execute step 5;

[0093] Step 5: Select the longest draft word sequence from the verified draft word sequences (the verified draft word sequence in the present invention means that each word in the sequence is the same as the word with the highest probability output by the large model in each step), and the selected draft word sequence is the inference result of the large model.

[0094] The other steps and parameters are the same as those in Specific Embodiments 7 to 9.

[0095] Example

[0096] The following example is used to explain the situation in detail. Assume that the current adjacency matrix has been initialized according to the hot start strategy, and the model has performed some decoding processes and made some updates to the adjacency matrix. The current context is "... volunteering as a guest". Figure 3As shown in the figure, we first select the word guest as the root node of the draft tree. Let's assume that the maximum depth of the draft tree is three layers. According to the current adjacency matrix, the corresponding candidate words are speaker, speak, and Spe. According to the pre-set tree structure, we select the top two candidate words speaker and speak as its child nodes here, so that the second layer of the draft tree is constructed. Then, we continue to search for speaker in the adjacency matrix, and select the top two words at and for as the child nodes of speaker; for speak, since it is ranked later in the second layer, we only select the word ings with the highest probability as its child node, so that the entire draft tree is constructed. After constructing the draft tree, we use the tree attention mechanism to calculate the corresponding mask matrix, compress all possible paths of the draft tree into a sequence, and send it to the large model for reasoning.

[0097] After inference, we extract all possible paths based on the draft tree and select the correct and longest path as the answer. The draft tree has 5 paths, namely guest speaker, guest speaker at, guest speaker for, guest speak, and guest speakings. According to the model calculation results, for guest, the word with the highest probability obtained by the model is speaker. Therefore, the latter two possible paths are excluded. Subsequently, for speaker, the word with the highest probability obtained by the model is at, so we select the correct and longest path guest speaker at as the answer.

[0098] While verifying the path, we also get the probability distribution corresponding to the draft words and the candidate words with the highest probability. For example, for the word speaker, the top two candidate words are at and is. We update the candidate word list corresponding to speaker in the adjacency matrix to at, is, and event. The same process applies to other draft words. In this way, we have completed the update of the entire adjacency matrix. Finally, the output word a of at of guest speaker at is used as the root node of the new draft tree, and the above process is repeated until the reasoning is completed.

[0099] We experimentally validate large models with three parameter sizes on two test sets, SpecBench and MBPP.

[0100] Table 1

[0101]

[0102] The SpecBench test set covers a variety of scenarios, including multi-turn dialogue, translation, summary, question answering, mathematical reasoning, and retrieval enhancement generation tasks, while the MBPP test set is a commonly used test set in the field of code generation. Our experimental results are shown in the ours row in Table 1, and the data of the best method in each task is bolded. The experimental results show that our method can achieve more than twice the acceleration effect, consistently outperforming other existing methods.

[0103] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A large model reasoning method based on adjacency matrix and speculative decoding, characterized in that: The method specifically comprises the following steps: Step 1: Select the first k candidate words for each word in the vocabulary, and construct an adjacency matrix M for storing the k candidate words of each word. The dimension of M is in, is the number of words in the vocabulary; Step 2: According to the last word in the input query sequence, a draft tree consisting of a draft word sequence is retrieved from the adjacency matrix M; Step 3: Use the nodes in the draft tree to construct a compressed sequence, and construct an attention mask matrix based on the draft tree; Step 4: Input the attention mask matrix and each draft word sequence in the compressed sequence into the large model, use the output of the large model to verify each draft word sequence in the draft tree, and update the adjacency matrix; After entering a new query sequence, return to step 2.

2. The large model inference method based on adjacency matrix and speculative decoding according to claim 1, characterized in that: The adjacency matrix M is initialized according to common expressions and common patterns in various queries.

3. The large model inference method based on adjacency matrix and speculative decoding according to claim 1, characterized in that: In the step 2, the draft word sequence retrieved from the adjacency matrix M is specifically: The last word in the input query sequence is recorded as x0, and any draft word sequence retrieved is recorded as (x0, x1, x2, ..., x i-1 ), i is the number of levels of the draft tree; The draft word sequence (x0, x1, x2, ..., x i-1 ) satisfies: In the adjacency matrix M, x1 is any one of the first k candidate words of x0, x2 is any one of the first k1 candidate words of x1, and x i-1 is x i-2 The first k i-2 Any one of the candidate words, and k≥k1…≥k i-2 .

4. The large model inference method based on adjacency matrix and speculative decoding according to claim 1, characterized in that: In step 3, a compression sequence is constructed using the nodes in the draft tree. The specific process is as follows: The root node of the draft tree is taken as the first word in the compressed sequence. In the compressed sequence, the second-level nodes of the draft tree are located after the first word, and the third-level nodes of the draft tree are located after the second-level nodes, until all nodes in the draft tree are traversed to obtain a compressed sequence.

5. The large model inference method based on adjacency matrix and speculative decoding according to claim 4, characterized in that: The attention mask matrix is ​​constructed based on the scratch tree, and the attention mask matrix satisfies: (1) All elements on the diagonal of the attention mask matrix are 1; (2) If the lth word in the compressed sequence and the l'th word in the compressed sequence are in the same draft word sequence, and the lth word does not precede the l'th word in the draft word sequence, then the element in the lth row and l'th column of the attention mask matrix is ​​1; l = 1, 2, ..., L, l' = 1, 2, ..., L, where L is the number of words in the compressed sequence; (3) Except for the elements with values ​​1 determined in (1) and (2), all other elements in the attention mask matrix are 0.

6. The large model reasoning method based on adjacency matrix and speculative decoding according to claim 1, characterized in that: The output of the large model is used to verify each draft word sequence in the draft tree and update the adjacency matrix; The specific process is: Step 1: record the last word in the input query sequence as x0, the large model outputs the first k candidate words of word x0, and the candidate word with the highest probability is recorded as x'1. Use the first k candidate words of word x0 output by the large model to update the candidate words of word x0 in the adjacency matrix, and continue to verify each draft word sequence whose second word is x'1 in the draft tree; Step 2, initialize a=1; Step 3: The large model outputs word x' a The first k candidate words of , and the candidate word with the highest probability is recorded as x' a+1 , using the word x' output by the large model a The first k candidate words for word x' in the adjacency matrix a The candidate words are updated, and the second word in the draft tree is x'1, the third word is x'2, ..., the a+2th word is x' a+1 The individual draft word sequences continue to be verified; Step 4, let a=a+1, return to step 3, until all draft word sequences are verified, and then execute step 5; Step 5: Select the longest draft word sequence from the verified draft word sequences. The selected draft word sequence is the inference result of the large model.

7. A large model reasoning system based on adjacency matrix and speculative decoding, characterized in that: The system includes a candidate word storage module, a draft tree generation module, an attention mask matrix generation module and a large model reasoning module; The candidate word storage module is used to store the first k candidate words of each word in the vocabulary in the adjacency matrix M; The draft tree generation module is used to retrieve a draft tree consisting of a draft word sequence from the adjacency matrix M according to the last word in the input query sequence; The draft word sequence retrieved from the adjacency matrix M is specifically: The last word in the input query sequence is recorded as x0, and any draft word sequence retrieved is recorded as (x0, x1, x2, ..., x i-1 ), i is the number of levels of the draft tree; The draft word sequence (x0, x1, x2, ..., x i-1 ) satisfies: In the adjacency matrix M, x1 is any one of the first k candidate words of x0, x2 is any one of the first k1 candidate words of x1, and x i-1 is x i-2 The first k i-2 Any one of the candidate words, and k≥k1…≥k i-2 ; The attention mask matrix generation module is used to construct a compressed sequence using nodes in the draft tree, and construct an attention mask matrix based on the draft tree; The large model reasoning module is used to verify each draft word sequence in the draft tree according to the input attention mask matrix and compression sequence, and update the adjacency matrix.

8. The large model reasoning system based on adjacency matrix and speculative decoding according to claim 7, characterized in that: The adjacency matrix M is initialized according to common expressions and common patterns in various queries.

9. The large model reasoning system based on adjacency matrix and speculative decoding according to claim 7, characterized in that: The working process of the attention mask matrix generation module is as follows: Step 1: Use the nodes in the draft tree to construct a compressed sequence; The root node of the draft tree is taken as the first word in the compressed sequence. In the compressed sequence, the second-level node of the draft tree is located after the first word, and the third-level node of the draft tree is located after the second-level node, until all nodes in the draft tree are traversed to obtain a compressed sequence; Step 2: Construct an attention mask matrix based on the draft tree. The attention mask matrix satisfies: (1) All elements on the diagonal of the attention mask matrix are 1; (2) If the lth word in the compressed sequence and the l'th word in the compressed sequence are in the same draft word sequence, and the lth word does not precede the l'th word in the draft word sequence, then the element in the lth row and l'th column of the attention mask matrix is ​​1; l = 1, 2, ..., K, l' = 1, 2, ..., L, where L is the number of words in the compressed sequence; (3) Except for the elements with values ​​1 determined in (1) and (2), all other elements in the attention mask matrix are 0.

10. The large model reasoning system based on adjacency matrix and speculative decoding according to claim 7, characterized in that: The working process of the large model reasoning module is as follows: Step 1: record the last word in the input query sequence as x0, the large model outputs the first k candidate words of word x0, and the candidate word with the highest probability is recorded as x'1. Use the first k candidate words of word x0 output by the large model to update the candidate words of word x0 in the adjacency matrix, and continue to verify each draft word sequence whose second word is x'1 in the draft tree; Step 2, initialize a=1; Step 3: The large model outputs word x' a The first k candidate words of , and the candidate word with the highest probability is recorded as x' a+1 , using the word x' output by the large model a The first k candidate words for word x' in the adjacency matrix a The candidate words are updated, and the second word in the draft tree is x'1, the third word is x'2, ..., the a+2th word is x' a+1 The individual draft word sequences continue to be verified; Step 4, let a=a+1, return to step 3, until all draft word sequences are verified, and then execute step 5; Step 5: Select the longest draft word sequence from the verified draft word sequences. The selected draft word sequence is the inference result of the large model.

Images