Big language model-based reasoning method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to the field of artificial intelligence, and discloses a reasoning method and device based on a large language model, electronic equipment and a storage medium. The method comprises the following steps: inputting a last token of a first token set into a prediction module, and outputting a second token set; carrying out parallel calculation on the second token set in a decoder layer, and outputting the next reasoning token of each token in the second token set; and inputting all the reasoning tokens into a prediction result decision device, for each prediction branch sequence in the second token set, matching the next reasoning token of the tokens in the sequence with the next token stage by stage from the first stage, and outputting the longest token sequence obtained by matching as a third token set by the prediction result decision device. Through a prediction-parallelization-judgment process, the problem of video memory bandwidth bottleneck caused by the fact that a large language model calculates tokens one by one is solved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence, and in particular to a reasoning method and device based on a large language model, an electronic device, and a storage medium. BACKGROUND

[0002] The large language model based on the transformer is a mainstream model architecture of generative artificial intelligence, such as GPT-4. Figure 1 The reasoning calculation of the large language model based on the transformer is performed in a manner of calculating and outputting each token (a basic unit of natural language text) one by one. This “Next Token” calculation manner determines that the calculation of the next token cannot be started until the calculation and output of the previous token are completed. At the same time, in order to pursue accuracy, the model parameters of the large language model are usually made larger and larger.

[0003] Therefore, the existing transformer architecture adopts a self-recurrent calculation manner of Next-Token of tokens. In each calculation cycle of generating a single token, these huge model parameters (in the form of a weight matrix) need to be completely loaded from off-chip memory to a computing chip. This makes the high-performance computing unit in a state of waiting for data loading most of the time, resulting in a serious waste of computing resources. SUMMARY

[0004] The purpose of embodiments of the present application is to provide a reasoning method and device based on a large language model, an electronic device, and a storage medium, so as to solve the problem that the self-recurrent calculation manner of Next-Token of tokens of the existing transformer architecture causes the computing unit to be in a state of waiting for data loading most of the time, resulting in a serious waste of computing resources.

[0005] To solve the above technical problems, an embodiment of the present application provides a reasoning method based on a large language model, comprising: inputting a last token of a first token set into a prediction module, and outputting a second token set; wherein the first token set is a continuous token sequence to be predicted, and the second token set contains a plurality of prediction branch sequences, each of which takes the last token of the first token set as a root node; performing parallel calculation on the second token set at a decoder layer, and outputting a next reasoning token of each token in the second token set; inputting all the reasoning tokens into a prediction result decision maker, and for each prediction branch sequence in the second token set, matching and confirming the next reasoning token and the next token of the token in the sequence step by step from the first level, and the prediction result decision maker outputs a longest token sequence obtained by matching as a third token set.

[0006] In the embodiment of the present application, through the process of "prediction-parallel-determination", the problem of memory bandwidth bottleneck caused by the traditional large language model due to the calculation of tokens one by one is solved. The method first generates a "second token set" containing multiple possible futures through the prediction module, and then performs one-time parallel calculation on the set at the decoder layer. This parallel processing mode combines the multiple independent model parameter loading processes required for generating multiple tokens into one, thereby greatly improving the parameter reuse rate and the utilization rate of computing resources, and effectively breaking through the limitation of memory bandwidth. Finally, the prediction result decision maker matches and confirms the next reasoning token and the next token of the token in the sequence step by step, and outputs the longest token sequence obtained by matching. Therefore, the method breaks the serial dependence of calculation, improves the output of single reasoning from a single token to a token sequence, and significantly speeds up the overall reasoning speed of the model.

[0007] Embodiments of the present application also provide a reasoning device based on a large language model, comprising: a prediction module configured to receive a last token of a first token set and output a second token set; wherein the first token set is a continuous token sequence to be predicted, and the second token set comprises a plurality of prediction branch sequences, each of which has the last token of the first token set as a root node; a decoder layer configured to accept the second token set and output a next reasoning token of each token in the second token set; and a prediction result decision device configured to receive all the reasoning tokens and match and confirm the next reasoning token and the next token of each token in each prediction branch sequence in the second token set from the first level to the last level, and finally output a longest token sequence as a third token set.

[0008] Embodiments of the present application also provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the reasoning method based on the large language model as described above.

[0009] Embodiments of the present application also provide a computer-readable storage medium storing a computer program, and the computer program is executed by a processor to implement the reasoning method based on the large language model as described above.

[0010] In addition, the second token set is calculated in parallel at the decoder layer, and the next inference token of each token in the second token set is output, including: loading model parameters and a first KV set from the video memory; the first KV set includes KV data of each token in the first token set, and / or KV data used to infer earlier tokens of the first token set; inputting the model parameters into a QKV parallel calculation module, performing parallel QKV calculation on the second token set to obtain a second KV set; the second KV set includes KV data of each token in the second token set; inputting the first KV set and the second KV set into a prediction parallel multi-head self-attention calculation module, performing parallel calculation on the second KV set, and outputting the next inference token of each token in the second token set. In the embodiment of the application, the method first loads the model parameters and the first KV set from the video memory at one time, and reads all the data that can be shared by the prediction tokens uniformly, which avoids the inefficient operation of repeatedly loading data in the traditional token-by-token calculation, and greatly improves the reuse rate of model parameters and historical KV data. Subsequently, the loaded model parameters are used to perform parallel QKV calculation on the second token set to obtain a second KV set, and the key QKV projection step is parallelized, so that the weight matrix can be shared by all prediction tokens in a batch after being loaded once. Finally, in the core self-attention calculation, the first KV set representing the historical context is combined with the second KV set newly generated in this round, and the final parallel calculation is performed based on this. This design optimizes the calculation data flow, maximally reduces the read and write times of the slow video memory, and thus can solve the problem of video memory bandwidth bottleneck.

[0011] In addition, the first KV set and the second KV set are input into a predicted parallel multi-head self-attention calculation module, the second KV set is calculated in parallel, and the next reasoning token of each token in the second token set is output, including: for each token in the second token set, selecting the KV vectors of all previous nodes in the corresponding branch sequence from the second KV set, inputting the first KV set into the predicted parallel multi-head self-attention calculation module, and outputting the next reasoning token of each token. In the embodiment of the present application, when performing parallel self-attention calculation, for each predicted token in the second token set, the system selects the KV vectors of all previous nodes in the corresponding branch sequence from the second KV set (i.e. the newly generated KV data in this round) and merges them with the first KV set. Thus, when each predicted token performs parallel self-attention calculation, its context environment not only contains complete historical information, but also contains correct path dependence information in the current prediction tree structure. This mechanism not only realizes efficient reuse of historical KV data, but also realizes immediate KV data reuse within a single calculation batch (for example, the KV data of a parent node can be reused by all its child nodes), further reducing data redundancy and calculation amount, making it possible to perform one-time, parallel and accurate reasoning verification on the entire complex prediction tree.

[0012] In addition, the method further comprises: after the second token set is subjected to parallel QKV calculation to obtain a second KV set, storing the second KV set in a QKV projection result temporary storage module; and after outputting a next inference token of each token in the second token set, selecting KV data corresponding to the third token set from the QKV projection result temporary storage module, and writing the corresponding KV data into the display memory according to the branch sequence of the third token set. In the embodiment of the application, the second KV set newly generated in parallel calculation is first stored in the temporary storage module in the calculation unit, instead of being directly written into the slow display memory outside the calculation unit. This step provides a fast data access channel for subsequent parallel self-attention calculation by avoiding frequent data exchange with the slow display memory during calculation, thereby improving the calculation efficiency. More importantly, the method delays the operation of updating the display memory until after the inference and decision are completed, and then selects the KV data corresponding to the third token set from the QKV projection result temporary storage module, and writes the corresponding KV data into the display memory according to the branch sequence of the third token set. The mechanism of "temporary storage first and then selection and writing" ensures that only the KV data that has been verified and confirmed as correct inference path is solidified into the KVCache representing long-term history, and invalid KV data generated by all incorrect prediction branches is directly discarded, thereby ensuring that the context history relied on by the next round of inference is accurate.

[0013] In addition, each token in the second token set has path index coding, and the path index coding is used to identify the level and parent node of the token in the corresponding branch sequence in the prediction result. The method comprises: based on the path index coding, selecting the KV data of the corresponding parent node of the token from the second KV set, merging the first KV set to input the prediction parallel multi-head self-attention calculation module, and outputting a next inference token of each token. In the embodiment of the application, by assigning a structured path index coding to each prediction token in the "second token set", the level and parent node of each token in the prediction branch tree are identified. Subsequently, when parallel self-attention calculation is performed, the calculation module can directly and quickly find the KV data of the parent node corresponding to each token in the second KV set according to the path index coding. Thus, the key problem of how to accurately and quickly dynamically combine the required context for each token when a large number of prediction tokens with complex hierarchical dependencies are processed in parallel is solved, and finally parallel and context-accurate inference of the entire prediction tree becomes possible. BRIEF DESCRIPTION OF DRAWINGS

[0014] One or more embodiments are illustrated by way of example in the drawings and are described herein in connection with the embodiments described. These embodiments are not intended to limit the scope of the embodiments to the embodiments described, but rather, serve as illustrations from which one skilled in the art can derive the general nature of embodiments. The drawings are not to scale and are intended for use only in conjunction with the description herein and the appended claims. The same or similar reference numbers in different drawings can indicate the same or similar components.

[0015] Figure 1 is a basic architecture of an existing inference model based on a large language model; Figure 2 is a detailed architecture of an existing inference model based on a large language model; Figure 3 is an inference architecture applied by an inference method based on a large language model provided by an embodiment of the present application; Figure 4 is a path index encoding structure diagram of a multi-branch multi-level Token prediction result output by a prediction module in an inference method based on a large language model provided by an embodiment of the present application; Figure 5 is an encoding diagram of a multi-branch multi-level Token prediction result output by a prediction module in an inference method based on a large language model provided by an embodiment of the present application; Figure 6 is an architecture diagram of a QKV projection parallel computing module in an inference method based on a large language model provided by an embodiment of the present application; Figure 7 is an architecture diagram of a prediction parallel multi-head self-attention computing module in an inference method based on a large language model provided by an embodiment of the present application; Figure 8 is a structure diagram of data organization in a QKV result temporary storage module in an inference method based on a large language model provided by an embodiment of the present application; Figure 9 is a work flow diagram of a prediction result decider module in an inference method based on a large language model provided by an embodiment of the present application; Figure 10 is a flow diagram of an inference method based on a large language model provided by an embodiment of the present application; Figure 11 is a flow diagram of a decoding stage in an inference method based on a large language model provided by an embodiment of the present application; Figure 12 is a flow diagram of parallel computing in a decoder layer in an inference method based on a large language model provided by an embodiment of the present application; Figure 13 is a structure diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0016] The Transformer-based large language model has become the mainstream model architecture of generative artificial intelligence. The basic idea is a self-attention mechanism based on token sequences, which infers the next token according to the current and previous token sequences, thereby realizing the sequence output of inference calculation. The current mainstream large language model adopts a decoder-only (only decoder) artificial intelligence architecture, which only uses the decoder architecture in the training and inference stages. The basic architecture is shown in Figure 1 The architecture is composed of a tokenizer, an embedding layer, an output layer, and multiple decoder layers. The input of the first decoder comes from the embedding vector output by the embedding layer, and the input of the remaining decoders comes from the output of the previous decoder layer. The output of the last decoder is sent to the output layer for output calculation of the token.

[0017] The detailed inference calculation architecture is shown in Figure 2 The global control signal mode controls the inference model to be in the prefill state or the decode state. In the prefill mode, the path selector selects the user input token processed by the tokenizer as the input of the embedding layer. In the decode mode, the path selector selects the output token of the previous round of inference calculation from the output layer as the input of the current round of inference.

[0018] Among them, the output layer and the multi-head self-attention network and FFN feedforward network in all decode layers mainly use matrix-vector multiplication and matrix multiplication. The coefficient matrix involved in the multiplication calculation is the model parameter. These parameters are the same for parallel calculation of different tokens in the same network architecture position, that is, they can be used simultaneously by multiple token vectors that can be parallel calculated. These model parameters are stored in the off-chip memory resource, and are loaded from the memory to the computing chip during calculation to complete the matrix multiplication and matrix-vector multiplication operation. Due to the huge amount of parameters, a very large memory bandwidth is required for each load. Therefore, in the single batch (single input token sequence) inference calculation, the decode stage only calculates one token at a time, and all model parameters need to be loaded once, which makes the memory bandwidth the bottleneck of the token output rate TPS (Token Per Second), greatly reducing the efficiency of the use of on-chip computing resources.

[0019] The workflow of this architecture mainly includes two stages: (1) The first is the prefill stage. When the tokens input from the embedding layer are all from user input, the system will be processed by the tokenizer and sent to the embedding layer for calculation. In this stage, all input tokens need to pass through multiple decoder layers for self-attention calculation and feedforward neural network calculation, and obtain their own K / V vector, which is used for subsequent self-attention calculation of tokens. This stage can process multiple tokens in parallel to share parameter matrices and improve computational efficiency.

[0020] (2) When all user input tokens are processed, the system enters the decode stage, that is, the stage of generating reply content one by one. In this stage, the tokens input into the embedding layer come from the output results of the previous inference calculation. Since self-attention calculation has a context-dependent relationship, the inference calculation of the K+1 token cannot be started until the K token is calculated, that is, each time only a single token can work in the mechanism of self-recurrence iteration.

[0021] This inherent serial dependency in the decoding stage fundamentally limits the parallelism of the calculation, and thus causes a serious performance bottleneck, especially in the single-user request (single batch, Single-Batch) scenario. This performance bottleneck is mainly caused by the following two technical defects related to the memory bandwidth: (1) Serious mismatch between calculation and memory bandwidth. The parameter size of large language models is extremely large, usually reaching hundreds of billions, and is stored in the form of weight matrices in off-chip memory. In the decoding stage, each token generated needs to load these huge model parameters from the memory to the computing chip completely. However, these parameters that are only used once cannot be loaded before the high-performance computing unit is idle and waiting, causing the system performance bottleneck not to be in the computing power, but in the memory bandwidth (memory-bound), resulting in a great waste of computing resources.

[0022] (2) KV Cache read-write overhead exacerbated by sequence growth. In self-attention computation, the model needs to read the Key (K) and Value (V) vectors of all historical tokens, which are stored in the form of KV Cache in the video memory. Specifically, self-attention computation refers to the correlation computation of the QKV vector of the current token with the K / V vector of all previous tokens in the sequence. In order to avoid repeated computation of the K / V vector of the historical token, these vectors are stored in the form of KV Cache in the video memory. This means that, for each new token generated, not only the model parameters need to be loaded, but also the entire KV Cache needs to be read from the video memory for self-attention computation, and the newly generated KV vector needs to be written back to the video memory after the computation is completed. With the growth of the generated sequence, the volume of the KV Cache increases linearly, and its read-write overhead further squeezes the already tight video memory bandwidth resources, making the performance bottleneck worse.

[0023] The above bottleneck is particularly significant in private deployment applications that focus on low latency and single-user experience. In such scenarios, it is usually not possible to amortize the data loading cost through batching. Therefore, the limitation of video memory bandwidth directly leads to a low key performance indicator, the number of tokens output per second (TPS), which not only seriously affects the user experience, but also makes the deployment and operation cost of large model inference computing power high.

[0024] To this end, the embodiments of the present application disclose a large language model-based inference method and device, which are used to solve the problem of severe waste of computing power resources caused by the fact that the computing unit is in a state of waiting for data loading most of the time due to the Next-Token autoregressive computation mode of the existing transformer architecture.

[0025] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. However, those skilled in the art can understand that, in the embodiments of the present application, many technical details are proposed in order to make the reader better understand the present application. However, the technical solutions claimed by the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present application. The embodiments can be combined and referenced with each other without contradiction.

[0026] An embodiment of the present application relates to a reasoning method based on a large language model, which can be applied to an electronic device equipped with an artificial intelligence (AI) inference chip or an acceleration card such as a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. Specifically, the execution subject can be a server, a data center, a high-performance computing platform, etc. installed with such an AI acceleration card, or other terminal devices embedded with an AI reasoning function.

[0027] The method of the embodiment of the present application comprises: inputting a last token of a first token set into a prediction module to output a second token set; wherein the first token set is a continuous token sequence to be predicted, and the second token set comprises a plurality of prediction branch sequences, each of which takes the last token of the first token set as a root node; performing parallel calculation on the second token set at a decoder layer to output a next reasoning token of each token in the second token set; inputting all reasoning tokens into a prediction result decision maker, for each prediction branch sequence in the second token set, matching the next reasoning token of a token in the sequence with a subsequent token step by step from a first level, and the prediction result decision maker outputs a longest token sequence obtained by matching as a third token set. In the embodiment of the present application, the process of "prediction-parallel-determination" solves the problem of memory bandwidth bottleneck caused by the traditional large language model due to the calculation of tokens one by one. The method firstly generates a "second token set" containing multiple possible futures by the prediction module, and then performs one-time parallel calculation on the set at the decoder layer. This parallel processing mode combines the multiple independent model parameter loading processes required for generating multiple tokens into one, thereby greatly improving the parameter reuse rate and the utilization rate of computing resources, and effectively breaking through the limitation of memory bandwidth. Finally, the prediction result decision maker matches the next reasoning token of a token in the sequence with a subsequent token step by step, and outputs a longest token sequence obtained by matching. Therefore, the method breaks the serial dependence of calculation, and improves the output of single reasoning from a single token to a token sequence, significantly accelerating the overall reasoning speed of the model.

[0028] The implementation details of the reasoning method based on a large language model of the embodiment of the present application will be specifically described below. The following content is only provided for the implementation details for easy understanding, and is not necessary for implementing the present solution.

[0029] The embodiment of the present application is applied to, for example Figure 3This is illustrated in a parallel inference computing device that supports multi-branch, multi-level prediction. The computing architecture is an optimized variant of the traditional Transformer large language model neural network architecture. Its core innovation lies in introducing a prediction module to generate multi-branch, multi-level future token predictions, and using these prediction results to achieve parallel inference of multiple predicted tokens in a single iteration of the decoder computation. This method can compute and output multiple consecutive tokens at once, thereby significantly increasing the model's token output capability.

[0030] like Figure 3 As shown, this computing architecture enhances the functionality of the traditional Transformer and mainly consists of the following core components: 1. Basic Computation Layer: This layer consists of one input embedding layer, one output layer, and multiple (L layers) identical decoder layers, which together perform the main inference computation tasks. The input of the first decoder layer comes from the output vector of the input embedding layer, and the input of each subsequent decoder layer comes from the output of the previous decoder layer. The computation result of the last decoder layer is fed into the output layer.

[0031] 2. New Key Control Modules: Path Selector: Based on the system's current operating mode, selects whether to send the "user-input token" or the "token generated by the prediction module" to the input embedding layer. Prediction Result Decision Controller: Responsible for judging the accuracy of the inference results after parallel computation, matching and identifying the successfully predicted token sequence, and outputting a KV_update signal. KV Cache Append Update Module: Based on the KV_update signal output by the prediction result decision controller, it accurately selects the KV projection vector corresponding to the correctly predicted token from the QKV projection result temporary storage module and appends them to the KV Cache storage area in the video memory.

[0032] The architecture switches between two operating modes via a global control signal: 1. Prefill mode: When mode = prefill, the system is in the prefill stage. At this time, the channel selector directly selects the input token data from the user to send into the calculation process, while the prediction module and the prediction result decision maker are in bypass state and do not participate in the work. In this mode, the token data format representing the user input is ({Itoken, Δpos}, p0), where Itoken is the value of the token, Δpos is the relative position offset, and p0 is the absolute position of the first token in the input token set {Itoken, Δpos} in the entire input token sequence. The system can process multiple such tokens in parallel to fully utilize the computing resources.

[0033] 2. Decode mode: When mode = decode, the system enters the core acceleration decoding stage. At this time, the prediction module and the prediction result decision maker are activated. The channel selector selects the predicted token data output by the prediction module to send into the input embedding layer for parallel calculation. The data format of these predicted tokens is ({Ptoken, idx, Δpos}, p0), where Ptoken is the predicted token value, idx is the code used to uniquely identify the path of the token in the prediction branch tree, Δpos represents the predicted level, i.e., the position offset, and p0 is the absolute position of the root token in the prediction branch tree, i.e., the last token in the input sequence set output by the previous round of inference of the prediction module, in the entire token sequence sent into the input embedding layer. Figure 4 and Figure 5 The encoding method of this branch path is disclosed in detail.

[0034] In terms of architecture implementation, all L decoder layers share the same design, but each layer uses different model parameters stored in the video memory obtained from model training when calculating. Each decoder layer is integrated with a complete calculation and cache management module, including: a self-attention normalization layer module, a QKV projection parallel calculation module, a predicted multi-head self-attention calculation module, a self-attention residual connection module, an FFN normalization module, a feedforward network (FFN) module, an FFN residual connection module, a KV history cache reading module, and a QKV projection result temporary storage module. These modules work together to support efficient parallel self-attention calculation and feedforward network processing.

[0035] Inside each decoder layer, the computation process mainly revolves around two core sub-modules: multi-head self-attention module and feed-forward network (FFN) module. To ensure the stability of the computation and the deep training effect of the network, both sub-modules are equipped with normalization and residual connection operations.

[0036] (1) The self-attention computation process includes: pre-normalization, first, the input vector ht will be processed by a self-attention normalization module to generate a normalized vector hNt. This step is designed to prevent numerical overflow or gradient anomaly in the intermediate calculation process. Residual connection, then, the normalized vector hNt is added element by element with the vector at output by the predicted multi-head self-attention computation module to obtain aht=hNt+at. The design of this residual connection is crucial to solving the gradient vanishing problem in deep neural networks, which ensures that information can be more smoothly transmitted between network levels.

[0037] (2) The feed-forward network computation process includes: pre-normalization, the output aht of the self-attention computation process will first pass through an FFN normalization module to obtain a normalized vector ahNt. Residual connection, the vector ahNt is added to the output vector ffnt of the feed-forward network FFN module to produce the final output vector ut=ahNt+ffnt of the decoder layer.

[0038] As known to those skilled in the art, for the above normalization module, normalization algorithms such as layer normalization or root mean square layer normalization can be used, but are not limited to them. The actual data type, quantity, and calculation speed requirements can be selected according to actual inference data.

[0039] In general, the multi-level multi-branch parallel inference computation architecture proposed in the above embodiments of the present application, as shown in Figure 3 , its core functions are realized by seven key sub-modules. These modules together constitute a complete closed loop of "prediction-parallel computation-judgment". Each module will be described in detail as follows: Submodule 1—the prediction module—is the starting point of the entire acceleration process in this scheme. It is responsible for actively and speculatively generating possible future token sequences, providing the data foundation for subsequent parallel computing. The function of the prediction module is to perform multi-branch, multi-level token prediction. Its input is the last token (denoted as Tn+k) in the token sequence that has been verified as correct by the prediction result decision unit in the previous round of computation; that is, the last token of the first token set mentioned in this embodiment. Based on this input, the module generates a predicted token tree with Tn+k as the root node, containing multiple branches and multiple levels; that is, the second token set mentioned in this embodiment. The structured representation of the prediction result and the branch path encoding method are shown in the appendix. Figure 4 and Figure 5 As shown. To achieve efficient prediction, this module can employ various prediction algorithms, including but not limited to: model-based prediction, using a small, pre-trained inference model with a small number of parameters (such as a small Transformer model) for fast inference and prediction; statistical prediction, based on a statistical database constructed from a large-scale corpus, which contains pre-defined contextual dependencies and occurrence probabilities between tokens; and reinforcement learning-based prediction, employing reinforcement learning model algorithms to dynamically optimize the prediction strategy to improve the hit rate. This prediction module also provides a series of configurable parameters for performance trade-offs in different application scenarios. These parameters include the maximum number of prediction output tokens, the maximum number of prediction levels, the maximum number of branches per level, and the contextual correlation threshold. By adjusting these parameters, users can find the optimal balance between prediction hit rate, computational resource utilization efficiency, memory bandwidth usage, and computational energy consumption.

[0040] Submodule 2 – the QKV projection parallel computing module – is located at the very front of the self-attention computation process. Its core function is to transform (or "project") the normalized input vector ht from the previous step into the three key vectors required by the self-attention mechanism: the query vector, the key vector, and the value vector. Its detailed internal architecture is shown in the attached figure. Figure 6The module receives its corresponding normalized embedding vector ht as input for each predicted token in the current round of parallel processing. Then, it generates three independent projection vectors for each ht through a linear transformation (usually vector multiplication with a weight matrix). To let the model understand the positional relationship of tokens in the sequence, the generated initial query vector and key vector also need to be fused with the positional encoding information of the token, finally producing qt and kt vectors containing position information. Value vector vt usually does not contain position encoding. The key of this module lies in its "parallel" feature. In the decoding stage, tokens of all prediction branches are treated as a batch and processed simultaneously. Therefore, this module can perform QKV projection calculation for all tokens in the batch. This means that in the same decoder layer, the weight matrix (Wq, Wk, Wv) used for projection calculation only needs to be loaded from the video memory once, and can be shared by all tokens, thereby greatly improving the reuse efficiency of parameter matrices and significantly saving the video memory bandwidth. After the calculation is completed, the QKV projection vector group (qt, kt, vt) generated for each input token is immediately sent to the QKV projection result temporary storage module for caching, so as to be efficiently read and used by the subsequent prediction parallel multi-head self-attention calculation module.

[0041] Submodule 3 - Prediction Parallel Multi-Head Self-Attention Calculation (PPMHA) module, is the place to perform the core self-attention operation, its detailed internal architecture is as shown in FIG. 3. Figure 7 The main task of it is to calculate the attention score between the Q vector of each predicted token in the current round of parallel processing and the KV vector of all previous tokens, and generate the final context vector after weighted by Softmax. In order to complete this calculation, two types of data need to be prepared efficiently for each token: 1. The QKV vector of the current token itself: that is, qt, kt, vt. These data are just generated by the previous "QKV projection parallel calculation module", so they can be directly read from the high-speed QKV projection result temporary storage module.

[0042] 2. KV history vector of all previous tokens: This is a core optimization of the architecture. The full KV history is dynamically constructed by concatenating data from two sources: (1) the history part from the GPU memory (i.e., the first KV set in the embodiment), which corresponds to all the long-term context tokens (denoted as k1~t-1 and v1~t-1) that have been verified and confirmed in all previous rounds. This part of data is loaded from the external GPU memory by the KV history cache reading module; (2) the immediate part from the temporary storage module (i.e., the second KV set in the embodiment), which corresponds to the parent node tokens on the current token prediction path in the current round. Their KV vectors are also generated in the current round. Therefore, this part of data is directly read from the high-speed QKV projection result temporary storage module without accessing the slow GPU memory.

[0043] This dual-source data loading strategy of "reading history data from GPU memory and reading immediate data from temporary storage" greatly optimizes the computing process. It ensures that the dependency relationships within the prediction branch can be quickly resolved in the high-speed on-chip cache, minimizing the number of read and write operations to the GPU memory, thereby significantly reducing the bandwidth bottleneck. After the data is prepared, the PPMHA module performs parallel multi-head self-attention computation and outputs a self-attention mapping vector at for each input prediction token.

[0044] Submodule 4 - KV history cache reading module, responsible for reading and loading historical KV Cache data from external GPU memory. This part of read data is the "long-term memory" of the entire token sequence, containing the K (key) and V (value) vectors corresponding to all processed and verified tokens since the start of computation. These historical KV data are continuously accumulated and written to the GPU memory by the KV cache append update module in the following two stages: pre-filling stage, generating KV vectors when processing the initial input sequence of the user; decoding stage, generating KV vectors for tokens verified as correct by the prediction result judge in each previous computing round. In the current parallel computing process, the historical KV Cache read by this module will be used as a shared and common context basis for the PPMHA module.

[0045] Submodule 5 - QKV projection result temporary storage module, acts as a high-speed on-chip "working area" or "temporary storage buffer". Its core function is to temporarily cache all newly generated QKV projection vector data in the current decoder layer during a single parallel computing round, thereby avoiding frequent data exchange with slow external memory during the speculation calculation process. In order to manage the data flow efficiently, this temporary storage module is designed with three independent I / O channels: (1) Write channel, which is dedicated to receiving the output from the QKV projection parallel computing module. When the QKV projection vectors of all predicted tokens in the current round are calculated, the results will be written to this module for caching through this channel. (2) Read channel, which serves the prediction parallel multi-head self-attention computing module. When performing self-attention calculation, the data required by this module (e.g., the QKV vector of the current token itself, and the KV vector of its parent node on the prediction path) are quickly read through this channel. (3) Upload channel: This channel is connected to the KV cache append update module. Once a round of calculation is complete and the prediction result decision maker confirms the correct token sequence, only the key-value vectors corresponding to these "valid" tokens are read through this upload channel to update the long-term key-value cache in external video memory. To support these efficient read and write operations, the internal data organization structure of this module is specially designed, as shown in the attached diagram. Figure 8 As shown, data is first stored according to decoder layers; within each layer, it is further partitioned according to three vector types: Q, K, and V; within each partition, the data is finally stored contiguously in multi-head order. This hierarchical and contiguous storage layout greatly facilitates batch reading and writing of similar data and simplifies the reorganization operations required before data is sent to various computing units.

[0046] The submodule 6-KV cache append update module plays the role of "data submission" or "state solidification" in the entire calculation process. The core of this module is to selectively update the KV vectors generated in this round and verified as valid from the second KV set to the long-term KV Cache (i.e. the first KV set) located in the external video memory at the end of each round of inference calculation. This module is activated after the prediction result judge completes the judgment. It receives the KV_update signal from the judge, which accurately indicates which tokens belong to the final adopted and correctly predicted sequence, and provides the storage location index of the KV vectors of these "valid tokens" in the QKV projection result staging module. After receiving the KV_update signal, the module accurately reads out the kt and vt projection vectors corresponding to the valid token sequence according to the index provided by the signal through the upload channel of the QKV projection result staging module. Subsequently, it writes these verified KV vectors to the KV Cache buffer area of the video memory in their correct order in the sequence. This "judgment first, update later" mechanism ensures that only verified and correct context information is solidified into the long-term history, thereby ensuring that the context history relied on by the next round of prediction and inference calculation is accurate.

[0047] The submodule 7-prediction result judge module is the end point of the entire "prediction-parallel calculation-judgment" process, responsible for the final verification and decision of the results of this round of parallel calculation. The detailed judgment workflow is shown in the attached Figure 9 The core task of this module is to match the "prediction path" given by the prediction module with the "actual result" obtained by parallel calculation level by level to find and output the longest valid token sequence. This module produces two outputs: 1. The final output token sequence, which is the final result provided to the user or downstream tasks. Its length is dynamically changing, depending on the hit situation of the prediction: (1) basic case (no hit or hit 0 level), if the first level prediction is all missed, or the prediction module does not output any prediction. This module will fall back to the traditional mode, outputting only one token (Tn+1), which is the direct inference result of the last final token Tn after this round of calculation. (2) Acceleration case (hit multiple levels): if the prediction is hit consecutively for k-1 levels, it means that the prediction from the first level to the k-1 level is consistent with the actual calculation result. In this case, this module can output a continuous sequence of k tokens (Tn+1, Tn+2,..., Tn+k) at once. This is equivalent to the output of the traditional Transformer architecture running k rounds in succession, achieving significant inference acceleration.

[0048] 2. KV_update control signal: This is an internal control signal, which is used to guide the work of the KV cache append update module. The signal contains the identification information (such as the storage location index in the temporary module) corresponding to the token sequence verified to be correct. According to this signal, the KV cache append update module can accurately know which newly generated KV vectors should be solidified into the long-term KV cache of the video memory.

[0049] In general, the inference method in the embodiment of the application can be applied to a large model inference architecture including the above-mentioned seven main modules, thereby realizing the complete closed-loop parallel inference of "prediction-parallel calculation-judgment".

[0050] It should be emphasized that the inference method of the embodiment of the application includes two nodes, a pre-filling stage and a decoding stage, and the complete multi-branch multi-level prediction inference calculation operation flowchart of the embodiment of the application is as shown in Figure 10 .

[0051] As shown in Figure 10 , it is an implementation step of an inference method based on a large language model according to an embodiment of the application, which includes a pre-filling stage in the early stage and a decoding stage in the later stage. The inference method in the decoding stage is as shown in Figure 11 , including steps 110 to 140.

[0052] In step 110, the last token of the first token set is input into the prediction module, and the second token set is output.

[0053] It should be noted that the first token set is a continuous token sequence to be predicted. In other words, the first token sequence can have two types, one is the sequence output by the decoder layer inference calculation and the prediction result decision after user input, and the other is the sequence output by the decoder layer inference calculation and the prediction result decision after several rounds of inference have been completed. The second token set contains several prediction branch sequences, and each prediction branch sequence has the last token of the first token set as the root node.

[0054] In the embodiment of the application, the prediction module used in the above-mentioned prediction stage is configured with a statistical database or a pre-trained inference model; wherein the statistical database is composed of several tokens with a preset context correlation relationship.

[0055] Specifically, as described above for the inference architecture, the prediction module can be implemented using different prediction algorithms, including using an inference model with a small number of parameters, a statistical database constructed based on token context dependency relationships, a reinforcement learning model algorithm, etc. The prediction module can also control the number of predicted tokens and hit efficiency by setting parameters, including the maximum number of predicted tokens, the maximum number of prediction levels, the maximum number of branches per level, the context correlation threshold per level, etc., in order to balance the prediction hit rate, computational resource usage efficiency, memory bandwidth, and computational energy consumption.

[0056] In step 120, the second token set is calculated in parallel at the decoder layer, and the next inference token for each token in the second token set is output.

[0057] In an embodiment of the present application, as shown in Figure 12 The above step 120 specifically includes steps 121 to 123: Step 121: load the model parameters and the first KV set from the memory; the first KV set includes KV data for each token in the first token set and / or KV data for earlier tokens used to infer the first token set.

[0058] Step 122: input the model parameters into the QKV parallel calculation module to perform parallel QKV calculation on the second token set to obtain the second KV set; the second KV set includes KV data for each token in the second token set.

[0059] Step 123: input the first KV set and the second KV set into the prediction parallel multi-head self-attention calculation module to perform parallel calculation on the second KV set, and output the next inference token for each token in the second token set.

[0060] It should be noted that the first KV set includes KV data for each token in the first token set and KV data for earlier tokens used to infer the first token set, and the second KV set includes KV data for each token in the second token set.

[0061] In a specific embodiment, in the above steps, for each token in the second token set, the KV vectors of all previous nodes in the corresponding branch sequence are selected from the second KV set, combined with the first KV set, and input into the prediction parallel multi-head self-attention calculation module to output the next inference token for each token.

[0062] As can be appreciated by those skilled in the art, in the specific embodiment described above, when performing parallel self-attention calculation, for each predicted token in the second token set, the system selects the KV vectors of all previous nodes in the corresponding branch sequence from within the second KV set (i.e., the newly generated KV data in this round) and merges them with the first KV set. Thus, when performing parallel self-attention calculation, each predicted token has a context environment that contains both complete historical information and correct path-dependent information in the current prediction tree structure. This mechanism not only enables efficient reuse of historical KV data, but also enables immediate KV data reuse within a single calculation batch (e.g., the KV data of a parent node can be reused by all its child nodes), further reducing data redundancy and computational load, making it possible to perform one-time, parallel, and accurate inference verification on the entire complex prediction tree.

[0063] In a specific embodiment, in the above step, after performing parallel QKV calculation on the second token set to obtain the second KV set, the second KV set is stored in the QKV projection result temporary storage module; and after outputting the next inference token of each token in the second token set, the KV data corresponding to the third token set is selected from the QKV projection result temporary storage module and written to the video memory. That is, the KV data corresponding to the third token set is selected from the QKV projection result temporary storage module and written to the video memory to update the KV Cache in the branch sequence.

[0064] As can be appreciated by those skilled in the art, in the specific embodiment described above, the newly generated second KV set in parallel calculation is first stored in the high-speed temporary storage module in the calculation unit, rather than being directly written to the slow video memory outside the calculation unit. This step provides a fast data access channel for subsequent parallel self-attention calculation by avoiding frequent data exchange with the slow video memory during calculation, thereby improving calculation efficiency. More importantly, this method delays the operation of updating the video memory until after the inference and decision are completed, and then writes the KV data corresponding to the third token set from the QKV projection result temporary storage module to the video memory. This "temporary storage first, then selective writing" mechanism ensures that only the KV data that has been verified and confirmed as correct inference path is solidified into the KV Cache representing long-term history, while all invalid KV data generated by incorrect prediction branches is discarded, thereby ensuring that the context history relied upon by the next round of inference is accurate.

[0065] In one specific embodiment, each token in the second token set has a path index code, which is used to identify the level of the token in the corresponding branch sequence in the prediction result and its parent node. In other words, each token in the second token set has a corresponding path index code. Thus, the parallel inference process in the prediction parallel multi-head self-attention calculation module can be: based on the path index code, selecting the KV data of the corresponding parent node from the second KV set, merging with the first KV set to input the prediction parallel multi-head self-attention calculation module, and outputting the next inference token of each token.

[0066] As known to those skilled in the art, in the above specific embodiment, by assigning a structured path index code to each prediction token in the "second token set", the level of each token in the prediction branch tree and its parent node are identified. Subsequently, when performing parallel self-attention calculation, the calculation module can directly and quickly find the KV data of the parent node corresponding to each token in the second KV set according to the path index code. Thus, the key problem of how to accurately and quickly dynamically combine the required context for each token is solved when a large number of prediction tokens with complex hierarchical dependencies are processed in parallel. Ultimately, it makes it possible to perform parallel and context-accurate inference on the entire prediction tree.

[0067] Specifically, in step 120, first, in order to provide necessary input for subsequent self-attention calculation, the QKV projection parallel calculation module (submodule 2) will be activated. This module receives the second token set (i.e. all predicted tokens to be verified) and the QKV projection weight matrix corresponding to the current decoder layer loaded from the video memory, and generates the Q, K, V vectors of each token in the set in parallel. These newly generated QKV vectors are then written as a whole, i.e. the second KV set, to the high-speed QKV projection result temporary storage module (submodule 5). After the preparation is completed, the core parallel inference calculation is then unfolded within the decoder layer. The predicted parallel multi-head self-attention calculation module (submodule 3) and the feedforward network module work together to complete this task. During the calculation process, the PPMHA module dynamically constructs the complete context required for each token from two sources: on the one hand, the first KV set containing all historical verified tokens is read from the video memory by the KV history cache reading module (submodule 4); on the other hand, it reads the newly generated second KV set in this round from the QKV projection result temporary storage module. Based on these two sets and the model parameters loaded from the video memory, the PPMHA and FFN modules calculate the corresponding next inference token result for each token in the second token set. Finally, in order to ensure the accuracy of the context history in the next round of inference, a key cache update step will be performed after the inference calculation is completed and the result is judged. The KV cache append update module (submodule 6) will know which predicted branch is verified as correct (i.e. the third token set) according to the KV_update signal issued by the prediction result judge. According to this signal, this module will extract only the KV data corresponding to the correct branch through the upload channel of the QKV projection result temporary storage module, and write them to the long-term KVCache in the video memory for subsequent calculation reuse.

[0068] In summary, for the above step 120, those skilled in the art can know that, first, when processing a batch of prediction tokens, the method will load the model parameters and the first KV set (historical KV data) commonly dependent on the batch of tokens from the video memory once according to the needs of the current parallel computing step. This way avoids the repeated and inefficient reading of the same data in traditional token-by-token calculation, thereby significantly improving the data reuse efficiency. Subsequently, the loaded model parameters are used to perform parallel QKV calculation on the second token set to obtain the second KV set, and the key QKV projection step is parallelized, so that the weight matrix can be shared by all prediction tokens within a batch after being loaded once. Finally, in the core self-attention calculation, the first KV set representing the historical context is merged with the second KV set newly generated in this round, and the final parallel calculation is performed based on this. This design optimizes the calculation data flow, maximally reduces the number of read and write operations on the slow video memory, and thus can solve the problem of video memory bandwidth bottleneck.

[0069] In step 130, all inference tokens are input into the prediction result decision maker. For each prediction branch sequence in the second token set, the next inference token and the next token of the token in the sequence are matched and confirmed level by level from the first level, and the prediction result decision maker outputs the longest token sequence obtained by matching as the third token set.

[0070] Specifically, the above step 130 is performed by the prediction result decision maker module (submodule 7), and the detailed decision workflow is as shown in the attached Figure 9The prediction result decision maker receives all the inference token results outputted by the current round of parallel computation and compares them with the initial predicted token set (i.e. the second token set). The specific matching process includes: first, in step 131, for each predicted branch sequence in the second token set, the decision maker starts matching from the first level (i.e. the direct child nodes of the root node). Specifically, it takes the next inference token (i.e. T(i)) actually calculated by the parent node token (e.g. P(i)) in the branch through step 120, and compares it with the predicted value (e.g. P(j)) of the child node token corresponding to the parent node. If they are completely consistent, the prediction of this level is considered a hit, and the verification process will continue down the branch; if they are not consistent, the child node and all subsequent branches will be immediately determined as prediction failure and discarded. Then, in step 132, after the matching process of all branches is completed, the prediction result decision maker will select the longest token sequence that matches successfully as the final output, i.e. the third token set. This output third token set can be directly delivered to the user or used for the next round of prediction. At the same time, the decision maker will also generate a key internal control signal KV_update, which contains the position index information of all tokens in the third token set, and send it to the KV cache append update module (sub-module 6) to ensure that only verified KV data is written to the long-term cache.

[0071] For step 130 described above, those skilled in the art can know that, by defining the second token set as "several predicted branch sequences" starting from the same "root node", the subsequent matching operation can be verified "level by level" along each predicted branch. That is, the "next inference token" actually calculated by a token in the sequence through the decoder is compared with the "next token" predicted by the branch. If they are consistent, it proves that the prediction of this level is a hit, and the verification process can continue to extend downward; if they are not consistent, the subsequent path of the branch is considered invalid. Finally, by selecting the "longest token sequence obtained by matching" as the final output, this scheme ensures the maximization of the benefits of single parallel computation.

[0072] In step 140, the KV data corresponding to the third token set is selected from the QKV projection result staging module, and the corresponding KV data is written to the video memory according to the branch sequence of the third token set.

[0073] Specifically, the step 140 is performed by a KV cache update module (submodule 6) responsible for solidifying the effective results of the current round of inference calculation and preparing for the next round of calculation. The execution basis of the module is the KV_update output signal from the prediction result decision maker, which indicates the storage location index of the third token set (i.e., "effective token") in the QKV projection result temporary storage module. According to the index provided by the KV_update signal, the module accurately reads out the KV (kt and vt) projection vectors corresponding to the third token set through the upload channel of the QKV projection result temporary storage module, and appends and updates these verified KV vector data to the KV Cache buffer area in the video memory.

[0074] For the step 140, those skilled in the art can know that in the parallel computing process, the QKV projection result temporary storage module will temporarily store the KV data generated by all prediction branches (including branches that are finally proved to be wrong). The present scheme "selects" only the KV data corresponding to the correct sequence after the correct third token set is determined, and "writes" it to the video memory. This "preliminary screening, then writing" operation ensures that only verified and effective KV data is appended to the historical KVCache, thereby ensuring that the context history relied on by the next round of inference is accurate.

[0075] It should be noted that in the embodiment of the present application, the second token set contains several prediction branch sequences, and each prediction branch sequence takes the last token of the first token set as the root node.

[0076] It should be noted that in order to effectively manage and utilize the hierarchical and dependent relationship of each prediction token in the "second token set", the embodiment of the present application adopts a tree-like encoding method, as shown in the attached Figure 4The prediction level and context relevance of each prediction token can be clearly expressed by a prediction path index encoding (IDX). The specific example of this encoding method is as follows: (1) Root node (0th level), the last token (i.e. Tn+k) output by the previous round of inference without prediction is taken as the root node of the prediction branch tree, and a basic path encoding is assigned to it, for example "000". (2) First level prediction: starting from the root node (IDX "000"), multiple (for example 4) possible next tokens are predicted. Assign them a unique path encoding, such as "100", "200", "300", "400". In the decision stage, if the actual calculation of the root node results in a next token that is exactly one of the four predictions, it means that the first level prediction has successfully hit on this branch. (3) Second level and subsequent prediction: continue to branch down based on the first level prediction. For example, starting from the token with IDX "100", two possible tokens can be further predicted, with encodings "110" and "120"; and starting from the token with IDX "200", four possible tokens can be predicted, with encodings "210", "220", "230" and "240". In this way, a multi-branch, multi-level prediction tree is formed.

[0077] Further, the attached Figure 5 The data organization method of this tree structure is specifically shown in the form of a list. Each row in the list corresponds to a prediction token and contains prediction token information and inference token information. The prediction token information includes: each prediction token (marked as P(i), including the root node token P(0) input to the prediction module first), its respective prediction path index encoding IDX(i), path position offset Δpos(i), and branch parent node encoding RIDX(i). The path position offset Δpos also identifies the prediction level, which can be directly calculated from the prediction path index encoding value IDX and the branch parent node encoding RIDX. Correspondingly, the inference token information includes: T(i) is the inference result next token calculated by the entire calculation structure for the corresponding i-th prediction token P(i); it is related to the self-attention calculation results of the branch parent node of the prediction token P(i) and all parent nodes of the parent node, all previous round inference results and all previous tokens of all user input tokens.

[0078] In addition, in the embodiment of the present application, the received user data also needs to be pre-filled before the decode stage, and the pre-filling process includes: comparing the number of tokens in the first token set with the number of parallel processing of the decoder layer; if the number of tokens is greater than the number of parallel processing, the first token set is equally divided into several segments, and the several segments are batch processed, and the generated KV data is written to the video memory.

[0079] Specifically, as shown in the pre-filling stage of the present application, Figure 10 Before entering the decode stage of the core, the system needs to pre-fill the received user initial input data (i.e. the first token set) first. The pre-filling process is carried out in the mode=prefill mode, at this time the path selector in the calculation architecture selects the input token data from the user to enter the input embedding layer, and the prediction module and the prediction result decision maker are in bypass state. In the pre-filling stage, all input tokens need to pass through multiple decoder layers for calculation to generate their respective K / V vectors, preparing for subsequent self-attention calculation. The specific way of pre-filling processing includes: first comparing the total number of tokens in the first token set with the upper limit of hardware parallel processing capability of the decoder layer. If the total number of tokens exceeds the upper limit, the first token set will be divided into several paragraphs of appropriate size, and then the paragraphs will be calculated one by one in the form of batch processing. After each segment processing is completed, the generated KV data will be written to the video memory by the KV cache update module. In this way, the KV data of the entire user input sequence is calculated and stored in the KV Cache, building a complete initial context history for the first round of decode stage.

[0080] In the embodiment of the present application, through the process of "prediction-parallel-determination", the problem of memory bandwidth bottleneck caused by the traditional large language model due to the calculation of tokens one by one is solved. This method first generates a "second token set" containing multiple possible future tokens through the prediction module, and then performs a one-time parallel calculation on the set in the decoder layer. This parallel processing method combines the multiple independent model parameter loading processes required for generating multiple tokens into one, thereby greatly improving the parameter reuse rate and the utilization rate of computing resources, and effectively breaking through the limitation of memory bandwidth. Finally, through the prediction result decision maker, the next reasoning token and the next token of the token in the sequence are matched level by level, and the longest token sequence obtained by matching is output. Therefore, this method breaks the serial dependence of calculation, and improves the output of single reasoning from a single token to a token sequence, significantly speeding up the overall reasoning speed of the model.

[0081] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two (including two), and similarly, "multiple levels" means more than two levels (including two levels), and "multiple branches" means more than two branches (including two branches).

[0082] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two), and similarly, "multiple levels" refers to more than two levels (including two levels), and "multiple branches" refers to more than two branches (including two branches).

[0083] The step division of the above method is only for clear description, and can be combined into one step or split into multiple steps in implementation, as long as the same logical relationship is included, all within the protection scope of the patent; adding insignificant modifications or introducing insignificant designs in the algorithm or process, but not changing the core design of the algorithm and process, are within the protection scope of the patent.

[0084] In addition, the examples mentioned in the above embodiments can be freely combined, and any combination can be understood as an embodiment. "Embodiment" or "example" appearing at various positions in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. Those skilled in the art can understand that the embodiments described herein can be combined with other embodiments.

[0085] Another embodiment of the present application relates to a reasoning device based on a large language model, comprising: a prediction module configured to receive a last token of a first token set and output a second token set; wherein the first token set is a continuous token sequence to be predicted, and the second token set comprises a plurality of prediction branch sequences, each of which has the last token of the first token set as a root node; a decoder layer configured to accept the second token set and output a next reasoning token of each token in the second token set; and a prediction result decision device configured to receive all reasoning tokens and match the next reasoning token of each token in each prediction branch sequence in the second token set with the next token from the first level, and finally output the longest token sequence obtained by matching as a third token set.

[0086] In some embodiments, the decoder layer is further configured to: load the model parameters and the first KV set from the video memory; the first KV set comprises KV data of each token in the first token set, and / or, the KV data used to infer the earlier tokens in the first token set, the QKV parallel computing module in the decoder layer is configured to receive the model parameters, and perform parallel QKV computation on the second token set to obtain a second KV set; the second KV set comprises KV data of each token in the second token set; and the prediction parallel multi-head self-attention computing module in the decoder layer is configured to receive the first KV set and the second KV set, and perform parallel computation on the second KV set to output the next inference token of each token in the second token set.

[0087] In some embodiments, the prediction parallel multi-head self-attention computing module is configured to, for each token in the second token set, select KV vectors of all previous nodes in the corresponding branch sequence from the second KV set, and perform inference computation by merging the first KV set to output the next inference token of each token.

[0088] In some embodiments, the QKV projection result temporary storage module is further configured to, after performing parallel QKV computation on the second token set to obtain the second KV set, store the second KV set; and after outputting the next inference token of each token in the second token set, select KV data corresponding to the third token set, and write the corresponding KV data into the video memory according to the branch sequence of the third token set.

[0089] In some embodiments, each token in the second token set has a path index code, and the path index code is used to identify the level and parent node of the token in the corresponding branch sequence in the prediction result; the prediction parallel multi-head self-attention computing module is further configured to, based on the path index code, select KV data of the corresponding parent node from the second KV set, and perform inference computation by merging the first KV set to output the next inference token of each token.

[0090] In some embodiments, the first KV set is obtained by pre-padding processing of the first token set; the pre-padding processing comprises: comparing the number of tokens in the first token set with the number of parallel processing in the decoder layer; if the number of tokens is greater than the number of parallel processing, equally dividing the first token set into several segments, and performing batch processing on the several segments, and writing the generated KV data into the video memory.

[0091] In some embodiments, the prediction module is configured with a statistical database or a pre-trained inference model; wherein the statistical database consists of several tokens with preset contextual relevance.

[0092] It is not difficult to see that this embodiment is a device embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.

[0093] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0094] Another embodiment of the present invention relates to an electronic device, such as Figure 13 As shown, it includes at least one processor 201; and a memory 202 communicatively connected to at least one processor 201; wherein the memory 202 stores instructions executable by at least one processor 201, the instructions being executed by at least one processor 201 to enable at least one processor 201 to perform the reasoning method based on the large language model as described above.

[0095] The memory 202 and processor 201 are connected via a bus, which may include any number of interconnecting buses and bridges, connecting various circuits of one or more processors 201 and memory 202 together. The bus may also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 201 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 201.

[0096] Processor 201 includes, but is not limited to, GPU and CPU, and is responsible for managing the bus and general processing. It can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 202 can be used to store data used by processor 201 during operation.

[0097] Another embodiment of the present application relates to a computer readable storage medium storing a computer program. The computer program, when executed by a processor, implements the method embodiments described above.

[0098] That is, those skilled in the art can understand that all or part of the steps of the method in the above embodiments can be completed by a program instructing relevant hardware, the program is stored in a storage medium, and includes a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0099] Those skilled in the art can understand that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A reasoning method based on a large language model, characterized in that, include: The last token of the first token set is input into the prediction module, and the second token set is output. The first token set is a continuous token sequence to be predicted, and the second token set contains several prediction branch sequences, each of which takes the last token of the first token set as its root node. The second set of tokens is computed in parallel at the decoder layer, and the next inference token for each token in the second set of tokens is output. All the inference tokens are input into the prediction result decision maker. For each prediction branch sequence in the second token set, starting from the first level, the next inference token of the token in the sequence is matched and confirmed with the next token. The prediction result decision maker outputs the longest matching token sequence as the third token set.

2. The reasoning method based on a large language model according to claim 1, characterized in that, The second set of tokens is computed in parallel at the decoder layer, and the next inference token for each token in the second set of tokens is output, including: Load model parameters and a first key-value set from video memory; the first key-value set includes the key-value data of each token in the first token set, and / or, for inference to obtain the key-value data of earlier tokens in the first token set; The model parameters are input into the QKV parallel computing module to perform parallel QKV computing on the second token set to obtain the second KV set; the second KV set includes the KV data of each token in the second token set; The first KV set and the second KV set are input into the prediction parallel multi-head self-attention calculation module, the second KV set is calculated in parallel, and the next inference token for each token in the second token set is output.

3. The reasoning method based on a large language model according to claim 2, characterized in that, The first key-value set and the second key-value set are input into the prediction parallel multi-head self-attention computation module. The second key-value set is processed in parallel, and the next inference token for each token in the second token set is output, including: For each token in the second token set, select the KV vectors of all preceding nodes in the corresponding branch sequence from the second KV set, merge them with the first KV set, input them into the prediction parallel multi-head self-attention calculation module, and output the next inference token for each token.

4. The reasoning method based on a large language model according to claim 2, characterized in that, include: After performing parallel QKV calculation on the second token set to obtain the second KV set, the second KV set is stored in the QKV projection result temporary storage module; After outputting the next inference token for each token in the second token set, the KV data corresponding to the third token set is selected from the QKV projection result temporary storage module, and the corresponding KV data is written to the display memory according to the branch sequence of the third token set.

5. The reasoning method based on a large language model according to claim 3, characterized in that, Each token in the second token set has a path index code, which is used to identify the level of the token in the corresponding branch sequence in the prediction result and its parent node; The method includes: selecting KV data of the corresponding parent node from the second KV set based on the path index encoding, merging it with the first KV set and inputting it into the prediction parallel multi-head self-attention calculation module, and outputting the next inference token for each token.

6. The reasoning method based on a large language model according to claim 2, characterized in that, The first KV set is obtained by pre-filling the first token set; The pre-filling process includes: Compare the number of tokens in the first token set with the number of parallel processes in the decoder layer; If the number of tokens is greater than the number of parallel processing operations, the first token set is divided into several segments, the segments are processed in batches, and the generated KV data is written to the video memory.

7. The reasoning method based on a large language model according to any one of claims 1 to 6, characterized in that, The prediction module is configured with a statistical database or a pre-trained inference model; The statistical database consists of several tokens with preset contextual relevance.

8. A reasoning device based on a large language model, characterized in that, include: The prediction module is used to receive the last token of the first token set and output the second token set; wherein the first token set is a continuous token sequence to be predicted, and the second token set contains several prediction branch sequences, each prediction branch sequence taking the last token of the first token set as the root node; The decoder layer is used to accept the second set of tokens and output the next inference token for each token in the second set of tokens; The prediction result decision unit is used to receive all the inference tokens and, for each prediction branch sequence in the second token set, to match the next inference token of the token in the sequence with the next token level by level, starting from the first level. Finally, it is used to output the longest matching token sequence as the third token set.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the reasoning method based on a large language model as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the reasoning method based on a large language model as described in any one of claims 1 to 7.

Images