Large language model reasoning method, device and equipment and computer storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a large language model reasoning method, device and equipment and a computer storage medium. The method comprises the following steps: acquiring an input sequence, a trained forward prompt embedding vector sequence and a trained backward completion embedding vector sequence; splicing an input embedding vector obtained by processing an input sequence through an embedding layer of a large language model with a forward prompt embedding vector sequence and a backward completion embedding vector sequence to obtain a combined input sequence; inputting the combined input sequence into a large language model, and forwards propagating the combined input sequence to at least one draft generation layer to obtain candidate draft output information corresponding to a forward prompt embedding vector sequence position; the draft generation layer is a Transform layer which is screened out from all Transform layers of a large language model and meets a reasoning precision condition and a reasoning speed condition based on a data set of a target reasoning task; and determining a reasoning result based on the confidence of the candidate draft output information, the reasoning result being the candidate draft output information, or submitting a token sequence obtained by sampling based on the candidate draft output information to all layers of the large language model for verification reasoning.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a large language model reasoning method and device, equipment and a computer storage medium. BACKGROUND

[0002] In recent years, large language models have shown broad application prospects in various fields due to their strong natural language understanding and generation capabilities. However, as the model size continues to expand, its deep network structure and high-dimensional parameter quantity characteristics result in a computationally intensive and resource-consuming reasoning process. When deployed on edge devices, the reasoning delay problem of large models is particularly prominent due to the limitations of computing power, memory and power consumption, which seriously restricts their application and popularization in real-time interaction and resource-constrained scenarios.

[0003] To address the above challenges, current research mainly focuses on layer pruning and intermediate layer output, draft model speculation sampling and self-speculation sampling. Methods based on layer pruning and intermediate layer output attempt to reduce overhead by terminating part of the layer computation in advance, but are prone to introducing semantic bias due to insufficient intermediate layer expression capability. Draft model-based speculation sampling methods generate candidate sequences through a small model and then verify them through the original model, which can reduce the number of large model calls, but face problems such as additional training costs, distribution differences and system complexity. Self-speculation sampling methods do not require the introduction of additional models, but still have obvious limitations in terms of draft quality, training difficulty and cache reuse. In addition, some adaptive scheduling-based solutions optimize the reasoning process at a macro level, but lack fine-grained adaptation to the internal structure of the model, making it difficult to achieve stable acceleration while ensuring generation quality.

[0004] Therefore, current acceleration methods are difficult to balance reasoning quality and reasoning speed. SUMMARY

[0005] The embodiments of the present application provide a large language model reasoning method, device, equipment and computer storage medium, which can balance the reasoning quality and reasoning speed.

[0006] In a first aspect, the embodiments of the present application provide a large language model reasoning method, which comprises: obtaining an input sequence, a trained forward prompt embedding vector sequence and a backward completion embedding vector sequence; concatenating the input embedding vector obtained by processing the input sequence through the embedding layer of the large language model with the forward prompt embedding vector sequence and the backward completion embedding vector sequence to obtain a combined input sequence; inputting the combined input sequence to the large language model and forward propagating to at least one draft generation layer to obtain candidate draft output information corresponding to the position of the forward prompt embedding vector sequence; the draft generation layer is a Transformer layer that meets the reasoning accuracy condition and the reasoning speed condition and is filtered from all Transformer layers of the large language model based on a data set of a target reasoning task; determining a reasoning result based on the confidence of the candidate draft output information, the reasoning result being the candidate draft output information or a result of submitting a token sequence sampled based on the candidate draft output information to all layers of the large language model for verification reasoning.

[0007] In a second aspect, an embodiment of the present application provides a large language model reasoning device, the device comprising: an acquisition module configured to acquire an input sequence, a trained forward prompt embedding vector sequence, and a backward completion embedding vector sequence; a splicing module configured to splice an input embedding vector obtained by processing the input sequence through an embedding layer of a large language model with the forward prompt embedding vector sequence and the backward completion embedding vector sequence to obtain a combined input sequence; an input module configured to input the combined input sequence to the large language model and forward propagate to at least one draft generation layer to obtain candidate draft output information corresponding to the position of the forward prompt embedding vector sequence; the draft generation layer is a Transformer layer that meets the reasoning accuracy condition and the reasoning speed condition and is filtered from all Transformer layers of the large language model based on a data set of a target reasoning task; a reasoning module configured to determine a reasoning result based on the confidence of the candidate draft output information, the reasoning result being the candidate draft output information or a result of submitting a token sequence sampled based on the candidate draft output information to all layers of the large language model for verification reasoning.

[0008] In a third aspect of the embodiments of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program.

[0009] In a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0010] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements any of the large language model reasoning methods described in the above embodiments.

[0011] The large language model inference method, apparatus, device, and computer storage medium of this invention acquire an input sequence, a trained forward cue embedding vector sequence, and a backward completion embedding vector sequence. The input embedding vector obtained by processing the input sequence through the embedding layer of the large language model is concatenated with the forward cue embedding vector sequence and the backward completion embedding vector sequence to obtain a combined input sequence. An enhanced context rich in task prior information is constructed at the model input end, injecting explicit generation guidance into the model without modifying the model's core parameters. The combined input sequence is input into the large language model and forward-propagated to at least one draft generation layer to obtain candidate draft output information corresponding to the position of the forward cue embedding vector sequence. The draft generation layer is a Transformer layer selected from all Transformer layers of the large language model based on the dataset of the target inference task, satisfying the inference accuracy and inference speed conditions. Through the draft generation layer, the model can output a high-quality, near-final intermediate representation before reaching the final layer, allowing most computations to terminate early at this intermediate layer. Based on the confidence level of the candidate draft output information, the inference result is determined. The inference result is either the candidate draft output information or the token sequence sampled from the candidate draft output information, which is submitted to all layers of the large language model for verification inference. This achieves adaptive acceleration of the inference process without requiring additional parallel model training or modification of the original model architecture. An evaluable fast generation path is established within the model, intelligently balancing inference efficiency and accuracy. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a large language model reasoning method provided in an embodiment of the present invention; Figure 2 This is one of the structural schematic diagrams of a large language model provided in an embodiment of the present invention; Figure 3 This is the second schematic diagram of the structure of a large language model provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of an embedded vector provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of an input sequence provided in an embodiment of the present invention; Figure 6 This is a flowchart illustrating another large language model reasoning method provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a large language model reasoning device provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present invention and not to limit the present invention. For those skilled in the art, the present invention can be practiced without some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present invention by illustrating examples of the invention.

[0015] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0016] To address the problems of existing technologies, embodiments of the present invention provide a method, apparatus, device, and computer storage medium for large language model reasoning.

[0017] The large language model reasoning method provided in the embodiments of the present invention will be introduced first below.

[0018] Figure 1 A flowchart illustrating a large language model reasoning method according to an embodiment of the present invention is shown. Figure 1 As shown, the method may include the following steps: Step 110: Obtain the input sequence, the trained forward cueing embedding vector sequence, and the backward completion embedding vector sequence; Step 120: The input embedding vector obtained by processing the input sequence through the embedding layer of the large language model is concatenated with the forward cueing embedding vector sequence and the backward completion embedding vector sequence to obtain the combined input sequence; Step 130: Input the combined input sequence into the large language model and propagate it forward to at least one draft generation layer to obtain candidate draft output information corresponding to the position of the forward cue embedding vector sequence; the draft generation layer is a Transformer layer selected from all Transformer layers of the large language model based on the dataset of the target reasoning task, which meets the reasoning accuracy and reasoning speed conditions. Step 140: Based on the confidence level of the candidate draft output information, determine the inference result. The inference result is either the candidate draft output information or the token sequence sampled based on the candidate draft output information is submitted to all layers of the large language model for verification inference.

[0019] The input sequence refers to the text prompts submitted by the user to the large language model. The forward cue embedding vector sequence is a set of learnable vectors that act as pre-injected generation guidelines highly relevant to the current task, guiding the model to generate high-quality candidate drafts. The backward completion embedding vector sequence is another set of learnable vectors that provides placeholder or completion context at specific positions in the input sequence, creating a more complete sequence environment for draft generation. The draft generation layer is a carefully selected intermediate layer from all Transformer layers of the large language model. This layer has the ability to generate intermediate results that are semantically and logically highly similar to the final output at a relatively fast speed. Confidence here specifically refers to a quantitative assessment of the reliability of the candidate draft output information generated by the model; the higher the confidence, the greater the likelihood that the output can be directly adopted.

[0020] Step 110: The input sequence is the original question or instruction provided by the user. The two embedding vector sequences are innovative components of this method. They are not fixed but learned through the preceding training process, and their parameters contain inductions about the output patterns of specific tasks. For example, in a code completion task, the forward cueing embedding vector sequence might learn the syntax structure of programming languages ​​and the calling patterns of commonly used APIs, while the backward completion embedding vector sequence might learn common ending patterns of function bodies or code blocks.

[0021] In step 120, the input sequence is converted into a dense vector representation, i.e., the input embedding vector, through the embedding layer of the large language model. Then, the input embedding vector, the forward cue embedding vector sequence, and the backward completion embedding vector sequence are concatenated in a preset order to form a complete combined input sequence. This concatenation operation mathematically expands the model's input context. By introducing these two task-adapted vector sequences, the model obtains richer task prior information and generation guidance signals beyond the original input text at the beginning of inference.

[0022] In step 130, the draft generation layer is a balance point selected from all layers of the model based on the dataset of the target inference task, achieving an optimal trade-off between generation speed and output accuracy. After propagation to this layer, the model outputs intermediate results corresponding to the positions of the forward cue embedding vector sequence; this is the candidate draft output information. The deep network of the Transformer model has powerful feature transformation and abstraction capabilities. The deeper intermediate layers can already capture the input semantics and form a preliminary, high-level representation close to the final output. For example, for a model with 40 layers, if the 28th layer is determined to be the draft generation layer, then this step only needs to compute the first 28 layers of the network to obtain a high-quality draft version, avoiding the computational overhead of the subsequent 12 layers.

[0023] In step 140, the final inference result is determined based on the confidence level of the candidate draft output information generated in step 130. The system evaluates the reliability of the draft. If its confidence level is high enough, it indicates that the draft is very close to the answer the model will ultimately provide, and it is directly adopted as the final inference result. Conversely, if the confidence level does not reach a preset threshold, it indicates that the certainty of the draft is insufficient and further verification is required. At this time, the system samples the output information of the candidate draft to obtain a specific token sequence, and then submits this sequence as input to the complete large language model, i.e., runs all Transformer layers for a rigorous verification inference, and uses the result of this verification as the final inference result. A dynamic decision-making mechanism is introduced. A quantifiable metric is used for careful evaluation. This enables on-demand allocation of computing resources: for simple, deterministic problems, the fast track is efficient; for complex, fuzzy problems, full computation is enabled to ensure quality.

[0024] In one embodiment, step 110 may specifically include the following steps: Obtain the training dataset for the target reasoning task; Based on the training dataset of the target reasoning task, determine the initial length of the backward completion embedding vector sequence and the initial length of the forward cue embedding vector sequence; Initialize the parameters of the forward hint embedding vector sequence and the backward completion embedding vector sequence; The original parameters of the large language model are frozen. Using the training dataset, the combined input sequence is used as the model input to train the forward cue embedding vector sequence and the backward completion embedding vector sequence until the preset convergence condition is met, and the trained forward cue embedding vector sequence and backward completion embedding vector sequence are obtained.

[0025] First, obtain the training dataset corresponding to the target inference task. Based on the analysis of this dataset, determine the initial lengths of the backward completion embedding vector sequence and the forward cue embedding vector sequence, respectively. The setting of this length is closely related to the characteristics of the task. For example, for a text summarizing task, the length of its backward completion embedding vector sequence may need to match the length range of common summaries, while the length of its forward cue embedding vector sequence may be proportional to the average length of the input document.

[0026] After determining the initial lengths of the backward completion embedding vector sequence and the forward cue embedding vector sequence, the parameters of these two vector sequences will be initialized. For example, the forward cue embedding vector sequence can be randomly initialized to preserve sufficient exploration space, while the backward completion embedding vector sequence can be zero-initialized to represent its initial state as a placeholder to be filled.

[0027] Freeze the original parameters of the large language model, that is, only use the forward cue embedding vector sequence and the backward completion embedding vector sequence as trainable parameters. This means that during training, the knowledge and capabilities of the large language model itself remain unchanged, and the entire purpose of training is to teach these two newly added vector sequences how to cooperate with the fixed model to efficiently generate high-quality drafts.

[0028] Specifically, to generate multiple sets of drafts simultaneously and with high quality, the forward embedding vector Epre and the backward embedding vector Epost are dynamically expanded along the sequence dimension of the input embedding vectors, with their lengths adaptively determined based on the task characteristics: This involves determining the length of the back-completion embedding vector sequence Epost. Epost is used to complete the truncated sequence of labels. Its length must match the maximum number of tokens output by the task. It calculates the maximum token length for each task's labeled content (label) in the target task training dataset, denoted as Epost. Manually preset Epost length threshold The final length of the Epost for this task is the minimum of the two. .

[0029] This involves determining the length of the forward cueing embedding vector sequence Epre. Epre is used to assist in draft generation, essentially acting as a generalized prompt. Its length is determined by combining the average length of the task input with a preset upper limit. The method is as follows: Calculate the average length of the tokens input for each task in the target task training dataset, denoted as... Set a length scaling factor α (α∈[0.1,0.3]), and manually preset the upper limit of Epre length. The final length of the Epre for this task is α× and The minimum value, .

[0030] During training, the model input is a combined input sequence constructed as described in step 120. By iteratively optimizing forward and backward propagation on this specific input format using the training dataset until the loss function converges to a preset condition, a forward cue embedding vector sequence and a backward completion embedding vector sequence are obtained that deeply understand task requirements and effectively guide draft generation. By introducing trainable parameters for these two vector sequences, the model can quickly adapt to specific task generation patterns and acceleration requirements without altering its core generalization capabilities.

[0031] In one embodiment, the steps of training the forward cue embedding vector sequence and the backward completion embedding vector sequence mentioned above may specifically include the following steps: During the fine-tuning training phase, the real labels of the training dataset are used as supervision signals for training; During the white-box knowledge distillation training phase, the output logic values ​​generated by the large language model on the training dataset are used as supervision signals for training. During training, the forward cue embedding vector sequence uses a causal masking mode in the self-attention calculation; the input embedding vector and the backward completion embedding vector sequence use a bidirectional masking mode in the self-attention calculation.

[0032] The fine-tuning training phase refers to using the true correct answers (real labels) from the task dataset to guide the model in learning basic task completion capabilities. The white-box knowledge distillation training phase is a special training method that uses the intermediate prediction distributions (output logistic values) generated by the original large language model on the same batch of data, without final selection, as a more refined supervisory signal. Causal masking patterns in self-attention computation are restrictive rules that ensure each position in a sequence can only focus on itself and its preceding positions, simulating the sequential process of word-by-word generation. In contrast, bidirectional masking patterns allow positions in a sequence to focus on all positions in the entire sequence, suitable for understanding complete contextual information.

[0033] The first phase of training is fine-tuning, using the dataset's real labels as supervision. This lays the foundation for task semantic understanding, enabling the model to directly learn the mapping from inputs with specific embeddings to the correct output. For example, in a machine translation task, the real labels are the standard target language translation. By minimizing the difference between its predictions and the standard translation, the model initially learns how to organize language to achieve translation within the context of introducing these two specific vector sequences. The goal of this phase is to give the newly added vector sequences basic task-completion capabilities, ensuring that their guiding direction is fundamentally correct.

[0034] Building upon fine-tuning training, the second stage employs white-box knowledge distillation training. This stage uses the output logistic values ​​generated by the original large language model on the same batch of training data as a supervisory signal for training. The aim is to ensure that the output of the draft generation layer, guided by the newly added vector sequences, approximates the target generated by the original large model after its full computation as closely as possible in terms of probability distribution. The output logistic values ​​of the original large model contain richer knowledge than a single true label, including probability assignments to different candidate words and information about word-to-word relationships. For example, when translating a sentence with multiple equivalent expressions, the true label might only provide one translation, while the logistic value output of the original large model would assign high probabilities to several correct synonyms. By learning this distribution through distillation, the draft generator not only learns to generate correct answers but also learns to mimic the original model's more nuanced thinking style and expressions of uncertainty, thus generating drafts that are statistically highly consistent with the final output.

[0035] Throughout the training process, the forward cue embedding vector sequence employs a causal masking pattern in the self-attention computation. This is because the positions corresponding to it need to be trained to perform the task of sequentially generating drafts, and the prediction at each position can only be based on the already generated content and the original input, ensuring consistency between the generation process during training and inference. On the other hand, the input embedding vector and the backward completion embedding vector sequence representing context completion employ a bidirectional masking pattern. This is because they represent known, complete contextual information; whether it's the user-input question or the supplementary background provided for generation, it should be fully cross-understood within the model. This design ensures that the model can fully and unhinderedly utilize all known information for reasoning when generating drafts.

[0036] Specifically, we first fine-tune training using the task dataset to initialize Epre and Epost with basic task adaptability; then, we further improve their ability to accurately fit the logits of large model outputs through white-box distillation training, avoiding the underfitting problem caused by direct distillation. The specific process is as follows: Fine-tuning Dataset Construction: This embodiment of the invention supplements the input embedding; therefore, the data needs to be processed at the embedding level. Specifically: a fine-tuning dataset is selected from the target task training set, covering all task types; each sample is concatenated according to the "prompt + label" standard format; after the sample passes through the embedding layer, a corresponding embedding vector is obtained, and a complete Epre is inserted before it. In its label part, Eposts are inserted at multiple different random positions to form a complete input embedding for training. The same processing is required for the output ground truth data: at the positions corresponding to the Eposts, real data corresponding to the length of the Epost is used to replace them to ensure that the loss calculation corresponding to the Epost part is performed during training.

[0037] Initializing Epre and Epost can include the following steps: Epre is initialized as a random vector following a normal distribution, and Epost is initialized as a vector of all zeros to reserve learning space for subsequent label fitting; its dimension is consistent with the model input embedding dimension. The attentionmask and lossmask are designed to correspond to the input embedding format designed in the above steps.

[0038] When calculating the attention mask on the QK pairs of the original data, it only focuses on the content of the original data and ignores the Epre part. However, when calculating the QK pairs corresponding to the Epost part, it needs to perform attention calculations on the Epre, the original data part, and the entire Epre part. Furthermore, because it is task-oriented training, the lengths of Epre and Epost are different for each task. During training, when the Epre and Epost parts of the attention mask participate in attention calculations, their lengths must correspond to the task. The loss mask is designed to only perform loss calculations on the Epost part, excluding other parts.

[0039] Epre and Epost training can specifically include the following steps: Freeze the original weights of the large model, and only use Epre and Epost as trainable parameters; each training batch contains only samples of the same task type and shares a single attention mask. The rest of the training process is the same as that of a regular large model training, and the Epre and Epost weight parameters are saved after training.

[0040] White-box distillation training aims to optimize the task compatibility of Epre and Epost, and may include the following steps: First, generate the original model's logits offline: input the training set of the target task into the complete large model (original model) for inference, obtain the logits of the entire input sequence, and save only the label sequence for subsequent dataset production.

[0041] Secondly, the distillation training dataset is constructed in the same way as in the fine-tuning training phase. The difference is that the ground truth value corresponding to Epost in the output part is the logits generated by the original model. The logits in other positions are set to arbitrary values ​​(such as 0) and do not participate in the loss calculation.

[0042] Training Epre and Epost may include the following steps: designing attention masks and loss masks; using positive KL divergence to evaluate the difference between the model output and the original model's logits when calculating the loss; iterative training until the model loss converges; and saving the final Epre and Epost weights for subsequent inference acceleration.

[0043] Fine-tuning training ensures the correctness of the generated drafts, resolving the issue of whether or not they exist. The subsequent white-box knowledge distillation training further optimizes the probability distribution characteristics of the drafts, aligning them closely with the mathematical expectation of the original large model's output distribution. This enhances the likelihood and safety of directly adopting drafts based on confidence in subsequent steps. The attention masking mechanism assigns correct information flow constraints to vector sequences with different functions, enabling the model to build a computational graph that conforms to the inherent logic of generative tasks during training. This training scheme, with extremely low parameter training costs, endows a fixed large language model with powerful, task-adaptive, and rapid draft generation capabilities, laying a core foundation for accelerating inference in subsequent dynamic decision-making.

[0044] In one embodiment, the following steps may be included before step 130: Select target sample batch data from the dataset of the target reasoning task; Input the target sample batch data into the large language model, obtain the output logic value of each Transformer layer in sequence, and record the inference time of each layer; Calculate the precision difference between the output logic value of each layer and the output logic value of the final layer, and calculate the inference speedup ratio of each layer relative to the final layer; From all Transformer layers whose accuracy difference is less than or equal to the preset accuracy loss threshold, select the Transformer layer with the largest inference speedup as the draft generation layer.

[0045] A representative batch of target sample data is selected from the dataset of the target inference task. This batch should cover the main data types and complexities of the task. Then, this batch of data is completely input into the large language model, and a full-network depth forward propagation is performed. During this process, the system sequentially extracts and saves the output logical values ​​of each Transformer layer of the model, while accurately recording the inference time consumed in completing the computation of each layer. The output logical values ​​are the unnormalized prediction vectors generated by the model after each network transformation; they represent the depth of understanding and predictive tendency of that layer regarding the input.

[0046] The precision difference between the output logic value of each layer and the final output logic value of the model is calculated. This difference quantifies the degree of deviation between the representation of the intermediate layer and the final complete model output, and is a core indicator for measuring whether it can maintain the quality of the result as a draft generation layer. Secondly, the inference speedup ratio of each layer relative to running all model layers is calculated. This ratio is obtained by dividing the inference time of the complete model by the inference time of this layer, which directly reflects the theoretical speed improvement that can be brought by using this layer.

[0047] Among all candidate Transformer layers with accuracy differences less than or equal to the threshold—that is, those layers that can guarantee generation quality no less than the set standard—the layer with the largest inference speedup is selected as the final draft generation layer. This is achieved through multi-objective optimization, maximizing inference speed while satisfying preset accuracy constraints. For example, in a text classification task, this process might reveal that layer 20 of the model outputs results highly similar to the final layer (e.g., layer 40) while being twice as fast. Therefore, layer 20 would be selected as the ideal draft generation layer for that task.

[0048] Specifically, this stage aims to select the top k layers from the Transformer layers of the large model that meet the accuracy constraints and have the fastest inference speed as draft generation layers. The specific process is as follows: From the training dataset for the target task, randomly select N samples, covering all task types with at least one sample for each task, to form a target sample batch. Input this target sample batch into the large model and obtain the output of each layer k (k ranges from 1 to the total number of layers L) in the Transformer layer hierarchy. Connect the output of this layer to the final output layer of the large model to obtain two key data points: The output logits of the k-th layer: denoted as , where b∈[1,B] (B is the batch size), t∈[1,T] (T is the sequence length), i∈[1,V] (V is the vocabulary size), represents the prediction logits of the b-th sample at sequence position t for the i-th token in the vocabulary when the k-th layer is cut off; Inference time at level k: Propagate each data point forward individually, record the time for each level, and finally take the average of all data at each level, denoted as . Simultaneously record the inference time of the complete model (L layers). .

[0049] Batch Mean Squared Error (MSE) Calculation: Measures the accuracy difference between the output of the k-th layer and the output of the complete model, as shown in the following formula: in, This is the final logits output for the complete model.

[0050] Speedup calculation: Measures the improvement in inference speed of the k-th layer relative to the complete model, as shown in the following formula: Set a maximum allowable accuracy loss threshold η, requiring the selected k to satisfy... Among all k that satisfy the precision constraint, the k with the largest speedup ratio S(k) (i.e. the fastest inference speed) is selected as the final draft generation layer.

[0051] By using quantification, the optimal balance between computational depth and inference speed within the model was found for a specific task. This avoids the problems that can arise from choosing the number of layers based on experience, such as being too conservative and resulting in limited speed improvement, or being too aggressive and damaging output quality. This selection process is task-adaptive; different tasks, due to their varying complexity, may select different optimal draft generation layers, demonstrating flexibility and refinement. This ensures that the draft generation pipeline, upon which subsequent acceleration processes rely, is built from the outset on optimal computational nodes that balance efficiency and fidelity, providing a fundamental guarantee for the effectiveness of the entire acceleration method.

[0052] In one embodiment, step 140 may specifically include the following steps: Step 210: Calculate the entropy value used to quantify confidence based on the probability distribution of each token position in the candidate draft output information; Step 220: If the entropy value of all positions is lower than the preset first entropy threshold, then the candidate draft output information is determined as the reasoning result. Step 230: If there is at least one position whose entropy value is higher than the first entropy threshold, then sample the candidate draft output information to obtain the token sequence; Step 240: Submit the token sequence to all layers of the large language model for verification reasoning to obtain the reasoning result.

[0053] Step 210 involves calculating the entropy value, used to quantify the confidence level, based on the probability distribution of each token position in the candidate draft output information. Entropy is a classic metric in information theory used to measure the uncertainty of random variables. In this context, the more concentrated the probability distribution of a token position, the lower its entropy value; the more even the distribution, the higher the entropy value. Therefore, the calculated entropy value directly and objectively reflects the model's confidence level when generating each draft word; the lower the entropy value, the higher the confidence level.

[0054] In step 220, the calculated entropy values ​​for all positions are compared with a preset first entropy threshold. This threshold is a key quality control parameter. If the comparison results show that the entropy values ​​for all positions are lower than this first entropy threshold, it means that the entire candidate draft output exhibits high certainty across all generation steps, and its overall confidence meets the criteria for direct adoption. In this case, no further calculations are performed, and the candidate draft output information is directly determined as the final inference result.

[0055] If, during the judgment in step 230, it is found that the entropy value at at least one location is higher than the first entropy threshold, it indicates that the certainty of the candidate draft is insufficient in one or more stages, and the overall confidence level does not meet the standard. In this case, direct adoption carries risks. The system will sample the output information of the candidate draft, for example, using methods such as temperature sampling or kernel sampling, to transform its probability distribution into a specific token sequence.

[0056] In step 240, to ensure the reliability of the final result, this specific token sequence is submitted to the full large language model, which involves calling all Transformer layers for a rigorous verification inference. The original large model uses this sequence as input, combining all its knowledge and computational depth to recalculate and output a result. This verification output will be used as the final, authoritative inference result. This process ensures that even if the draft confidence is insufficient, the final output is still guaranteed by the full capabilities of the model.

[0057] Specifically, based on the selected draft generation layer and the trained Epre and Epost, inference acceleration is achieved through "draft generation - adaptive verification - dynamic length adjustment". The specific process is as follows: Based on the current inference task type, load the corresponding Epost (length). With Epre (length) Weights. The input prompt is processed through the Embedding layer and then concatenated with Epost and Epre to form an "Epost + prompt embedding + Epre" structure. This structure is input into the large model and propagated to the draft generation layer k. A KV-Cache corresponding to the Epost and prompt embeddings during inference is saved for computational reuse in the subsequent validation phase. The output of the draft generation layer k is directly connected to the original model's output layer to obtain the draft logits corresponding to Epre. The entropy value of each draft logit is calculated. The entropy value is used to measure the uncertainty of the draft. The entropy value calculation formula is as follows: in The token probability is the result of Softmax normalization of the draft logits.

[0058] Directly accept drafts: After black-box distillation, the generated drafts already possess a certain level of confidence for the task. If the entropy value of all drafts is less than α (the confidence level meets the requirement), then the draft is directly used as the reasoning result, skipping subsequent verification and reducing computational load.

[0059] Further validate the draft: If the entropy value of a draft is greater than α (insufficient confidence), then generate draft tokens according to the preset sampling strategy (such as temperature sampling) and input them into the complete large model for validation.

[0060] Therefore, when the draft quality is extremely high, verification and output results can be completed through extremely low-cost entropy calculation and comparison, completely skipping the computation of subsequent heavy network layers and achieving maximum speedup benefits. When there is uncertainty in the output of the fast channel, it can automatically detect and switch to a rigorous but computationally more expensive verification mode, ensuring the consistency of the output results with the capabilities of the original model and preventing quality loss introduced by acceleration. This dynamic branching mechanism enables the entire system to make optimal real-time decisions between inference efficiency and output reliability based on the specific circumstances of each generation, thereby achieving the best balance between acceleration and fidelity overall.

[0061] In one embodiment, step 240 may specifically include the following steps: Reuse the key-value cache that was previously computed and saved before the draft generation layer, corresponding to the input embedding vector and the backward completion embedding vector sequence; Update the key-value cache with the key-value sequence of tokens; Starting from the layer below the draft generation layer, the updated key-value cache is used as the input for the self-attention computation of the remaining Transformer layers to perform confirmatory reasoning and output the reasoning results.

[0062] Key-value caching is a key technical component in the self-attention mechanism of the Transformer model. When processing a sequence, it calculates and stores a key vector and value vector for each sequence position, which is used to efficiently calculate attention scores in subsequent generation steps without recalculating key-value pairs of historical sequences.

[0063] The key-value cache corresponding to the input embedding vector and the backward completion embedding vector sequence, which was computed and saved during the draft generation process in step 130, is reused. This cache fully records all attention computation results for the original input and fixed context completion parts from the first layer of the model to the draft generation layer, representing the computational cost already paid. The newly generated, unverified token sequence is integrated into the existing computational context. To do this, the key-value pairs of the token sequence are updated to the key-value cache. This means that only the keys and values ​​from the first layer to the draft generation layer are computed for the newly added token sequence portion, and these newly computed key-value pairs are appended to the existing cache. This step only computes the newly added, relatively short token sequence, rather than recompiling the entire long sequence.

[0064] Based on the updated key-value cache, validating inference can be performed efficiently. Starting from the layer after the draft generation layer, the system uses this updated key-value cache as input for the self-attention computation of all remaining Transformer layers, continuing forward propagation. Since the key-value information of all historical and current sequences is fully stored in the cache, the self-attention computation of each remaining layer can directly utilize these caches for fast attention score calculation without needing to recalculate the output of previous layers. This process continues until all remaining layers are passed, and finally, the output layer produces the validated inference result.

[0065] Specifically, before the draft generation layer k, the "non-Epost KV-Cache" saved in step 2 is reused, and the KV of the newly generated draft tokens is updated to the cache; after the draft generation layer k, the calculation of the remaining Transformer layers continues, and the KV not in the cache is updated to the KV-Cache; based on the previous position token predicted by the original model, if the current draft token is consistent with the original model prediction, the draft is accepted; if the next draft token based on the current draft token is consistent with the original model prediction, it is accepted; if they are inconsistent, the draft is rejected and verification stops. Through the key-value cache reuse mechanism, the expensive computation of the original input and fixed context is completely saved.

[0066] Thus, through an incremental update mechanism, only the necessary computations are performed on the new draft sequence and integrated into the existing context. Ultimately, the computation of the remaining layers can be performed smoothly on a complete yet efficiently constructed cache. This design ensures that the computational overhead of verification reasoning is mainly concentrated on the length of the new draft sequence, rather than the length of the entire input sequence, thereby keeping the verification cost at a low level.

[0067] In one embodiment, after step 140, the following steps may also be included: Obtain feedback information on the candidate draft output information in this round of inference. The feedback information includes the entropy of the probability distribution of each token position in the candidate draft output information. Based on the feedback, adjust the generation length of the forward cue embedding vector sequence for the next inference.

[0068] By introducing a continuous learning feedback adjustment mechanism, the aim is to dynamically optimize the processing strategy for similar requests in the future based on the actual performance of the current generation. This involves adjusting the generation length of the forward cue embedding vector sequence used in the next inference. This mechanism first needs to obtain feedback information from the current inference round. Specifically, this information is the entropy sequence of the probability distribution of each token position in the candidate draft output information, which was used for decision-making in step 140. This entropy sequence objectively records the deterministic fluctuations of the model when generating each word in the draft, and is key data for evaluating whether the current preset draft length is reasonable.

[0069] The generation length of the forward cue embedding vector sequence during the next inference is adjusted based on this entropy value sequence. The trend of the entropy value sequence is used as a direct measure of the stability and difficulty of draft generation. The adjustment strategy follows an intuitive logic: if the entropy value increases sharply in the first half of the draft generation, it indicates that the task has become uncertain at that length, and the quality of subsequent generation may decrease. Therefore, the draft length should be shortened next time to focus on generating more certain and higher-quality initial content. Conversely, if the entropy value remains stable and low until the end of draft generation, it means that the current length is sufficient for the model, and a longer draft can be maintained or tried next time to pursue greater acceleration potential.

[0070] For example, in a dialogue generation task, if the system finds that the entropy value of the 8 draft words generated for the instruction "explain a concept" is consistently low, it may maintain or attempt to generate a 10-word draft the next time it encounters a similar instruction. However, if the entropy value increases significantly when the instruction "create a story ending" is generated by the 5th word, the draft length may be reduced to 4 words the next time to ensure the high confidence of the draft segment itself.

[0071] By coupling the entropy feedback of the current inference round with the draft length decision for the next inference round, the system can perform real-time optimization based on the dynamic characteristics of actual task instances. This allows the method to differentiate and respond to generation requirements of different complexities or even different specific inputs under the same task, achieving fine-grained, context-aware acceleration strategy adjustments. In the long run, this helps the system automatically converge its draft generation strategy to the most efficient balance point for specific task flows during continuous service, thereby continuously improving the overall average inference efficiency and resource utilization through continuous interaction, achieving a technological leap from static acceleration to dynamic self-optimization.

[0072] In one embodiment, the step of adjusting the generation length of the forward cue embedding vector sequence for the next inference based on feedback information may specifically include the following steps: Based on the feedback information, the entropy of each token position in the candidate draft output is calculated sequentially in this round of inference; If the entropy of the nth token position exceeds the preset second entropy threshold, then n will be used as the length of the forward cue embedding vector sequence for the next inference. If the entropy of all token positions does not exceed the second entropy threshold, then the current length of the forward hint embedding vector sequence is maintained during the next inference.

[0073] Based on the feedback information, namely the entropy of the probability distribution of each token position in the candidate draft output, each generation position of the draft sequence in this round of inference is sequentially traversed and analyzed. This sequential calculation and analysis simulates the logic of examining the stability of the generation process word by word.

[0074] A preset second entropy threshold is used as the critical point to determine whether the generation stability has undergone a qualitative change. The system performs conditional checks while sequentially checking the entropy value at each position. If, during the check, the entropy value corresponding to the nth token position exceeds the preset second entropy threshold, the algorithm immediately triggers an adjustment mechanism. Here, n represents the index of the first token where a significant decrease in determinism occurs. At this point, the system directly determines the value of n as the generation length of the forward cue embedding vector sequence for the next inference iteration.

[0075] All generated lengths prior to the first position with high uncertainty are considered safe and efficient generation intervals within the current task context. For example, if the entropy values ​​of the first three generated words are low, but the entropy value of the fourth word exceeds the second threshold, the system determines that "3" is a reliable generation length and sets the draft generation target to 3 tokens for the next similar request.

[0076] If the sequential checks are completed and the entropy values ​​of all token positions do not exceed the preset second entropy threshold, it indicates that the entire length of the draft used this time was completed with high confidence, and the current generation length setting is appropriate or even conservative. In this case, the system will not shorten the length, but will decide to maintain the current length of the forward cue embedding vector sequence in the next inference to continue exploring and utilizing the existing speedup potential.

[0077] By introducing a clear and quantifiable stability threshold, the adjustment process is automated and objective, avoiding subjective assumptions. The algorithm features real-time response, making adjustment decisions immediately upon detecting signs of increasing uncertainty, resulting in rapid response. Its adjustment strategy is inherently conservative, always prioritizing the quality of the draft segments and shortening their length to a range where determinism is guaranteed, fundamentally maintaining the reliability of the fast-track output. Simultaneously, the algorithm is also cumulatively optimized; through continuous monitoring and feedback, the system can adaptively learn and converge to a maximum, stable draft generation length for different task modes or input types, thereby progressively exploring and improving the efficiency ceiling of inference acceleration while ensuring a baseline of quality.

[0078] The large language model inference method, apparatus, device, and computer storage medium of this invention acquire an input sequence, a trained forward cue embedding vector sequence, and a backward completion embedding vector sequence. The input embedding vector obtained by processing the input sequence through the embedding layer of the large language model is concatenated with the forward cue embedding vector sequence and the backward completion embedding vector sequence to obtain a combined input sequence. An enhanced context rich in task prior information is constructed at the model input end, injecting explicit generation guidance into the model without modifying the model's core parameters. The combined input sequence is input into the large language model and forward-propagated to at least one draft generation layer to obtain candidate draft output information corresponding to the position of the forward cue embedding vector sequence. The draft generation layer is a Transformer layer selected from all Transformer layers of the large language model based on the dataset of the target inference task, satisfying the inference accuracy and inference speed conditions. Through the draft generation layer, the model can output a high-quality, near-final intermediate representation before reaching the final layer, allowing most computations to terminate early at this intermediate layer. Based on the confidence level of the candidate draft output information, the inference result is determined. The inference result is either the candidate draft output information or the token sequence sampled from the candidate draft output information, which is submitted to all layers of the large language model for verification inference. This achieves adaptive acceleration of the inference process without requiring additional parallel model training or modification of the original model architecture. An evaluable fast generation path is established within the model, intelligently balancing inference efficiency and accuracy.

[0079] The following example illustrates this, with the large language model deployed on a mobile phone. The open-source model llama 3.2-3b has 28 transformer layers. After fine-tuning training with intent recognition, SMS classification, and SMS summarization data, it possesses the capabilities of intent recognition, SMS classification, and SMS summarization. Inference on a mobile phone achieves a speed of only 2 tokens / second. The training data for intent recognition contains 30,000 data points, encompassing 355 intents, as shown in the example below: {"prompt": "As a smart assistant, please return the intent #intent# and slots #slots# based on the input, including the necessary parameters, presented in JSON format. Input:\n0.1 square kilometers converted to square feet","label":"{"intents": [{"intent": "convert area units", "slots": {"target area": ​​"square feet", "source area": ​​"square kilometers", "value": "0.1"}}]}"} The training data for SMS classification consists of 30,000 messages, encompassing six categories: user behavior notifications, other notifications, advertising SMS, fraudulent SMS, verification codes, and personal SMS. An example is shown below: {"prompt": "Please directly answer the following SMS category:\n

Tianwai

HR System

[0080] 2. Hierarchical Reasoning and Result Collection: Input the test batch into Llama3.2-3B, and sequentially obtain the output of each layer k in the order of Transformer layers 1 to 28, then connect it to the final output layer (lm_head layer) of the model, as follows. Figure 2 As shown: Collect two types of key data: The output logits of the k-th layer: denoted as Where b∈[1,128], t∈[1,64], i∈[1,128256], and record the token prediction logits for each sample and each sequence position in each layer; Inference time at level k: Each data point undergoes forward propagation independently, which is a profile process that records the forward propagation time at each level. (Unit: ms), average value of 128 data points, representing the inference time of the complete model (28 layers). .

[0081] 3. Calculation of accuracy and speed indicators: (1) Batch mean squared error (MSE) calculation: measures the accuracy difference between the output of the k-th layer and the output of the complete model; (2) Speedup calculation: measures the improvement in inference speed of the k-th layer relative to the complete model; In this embodiment, the data for the key layers are listed in Table 1: Table 1 A pre-defined maximum allowable accuracy loss threshold η=1 is set, requiring that the selected k=8 values ​​satisfy the following conditions. The speedup ratio S(k) is the largest (i.e. the reasoning speed is the fastest), so k=8 is used as the final draft layer.

[0082] II. Dynamic Expansion Phase of Embedding Vectors Adapted to the Task: Determining Task-Specific Epre and Epost To generate multiple sets of drafts simultaneously and with high quality, the forward embedding vector Epre and the backward embedding vector Epost are dynamically expanded along the sequence dimension of the input embedding vectors. The lengths of both are adaptively determined based on the task characteristics. The specific method is as follows: 1. Determining the length of the backward embedding vector Epost ( ) Epost is used to complete the truncated sequence after the label is removed. The length must match the maximum number of tokens output by the task. The method is as follows: In the training dataset for the statistical target task, determine the maximum token length of the labeled content (label) for each task, and the intent to identify. =55. SMS Classification =7. SMS Summary =20; Manually preset Epost length threshold =10; The final length of the Epost for each task is the minimum of the two values, which are 10, 7, and 10 respectively.

[0083] 2. Determining the length of the forward embedding vector Epre ( ) Epre is used to assist in draft generation, essentially acting as a generalized prompt. Its length is determined by combining the average length of the task input with a preset upper limit. The method is as follows: Calculate the average length of the tokens input for each task in the target task training dataset, denoted as... Set a length scaling factor α (α∈[0.1,0.3]), and manually preset the upper limit of Epre length. The final length of the Epre for this task is α× and The minimum value, In this embodiment, α = 0.3. =10, then intent recognition =10. SMS Classification =3. SMS Summary =6, specifically as follows Figure 3 As shown.

[0084] III. Fine-tuning Training + White-box Distillation Training Phase: Optimizing the draft fitting capabilities of Epre and Epost First, fine-tuning training is performed using the task dataset to initialize Epre and Epost, giving them basic task adaptability. Then, white-box distillation training is used to further improve their ability to accurately fit the logits of large model outputs, avoiding underfitting issues caused by direct distillation. The specific process is as follows: 1. Fine-tuning training: Initialize the task compatibility of Epre and Epost.

[0085] (1) Fine-tuning dataset construction: Unlike conventional training methods, this approach supplements the input embedding. Therefore, the data needs to be processed at the embedding level. Specifically, a fine-tuning dataset is selected from the target task training set, covering all task types. Each sample is concatenated according to the "prompt + label" standard format. After the sample passes through the embedding layer, the corresponding embedding vector is obtained. A complete Epre is inserted in front of it, and Epost is inserted at multiple different random positions in its label part to form a complete input embedding for training.

[0086] The same processing is required for the output ground truth data. At the position corresponding to Epost, replace it with the real data corresponding to the length of Epost to ensure that the loss calculation corresponding to the Epost part is performed during training.

[0087] The following is illustrative data; the actual data is in vector form. Input Embedding as follows Figure 4 As shown in (a), the truth value is as follows Figure 4 As shown in (b).

[0088] (2) Initialize Epre and Epost: Epre is initialized as a random vector following a normal distribution, and Epost is initialized as a vector of all zeros, reserving learning space for subsequent label fitting. Its dimension is consistent with the model input embedding dimension.

[0089] (3) Design of attention mask and loss mask: Corresponding to the input embedding format designed in (1): The attentionmask does not focus on attention calculations for the Epre part; when calculating on the QK pairs of the original data, it only focuses on the content of the original data, ignoring the Epre part; however, when calculating the QK pairs corresponding to the Epost part, attention calculations are required for the Epre, the original data part, and the entire Epre part. Furthermore, because training is task-oriented, the lengths of Epre and Epost differ for each task. During training, when the Epre and Epost parts of the attentionmask participate in attention calculations, their lengths must correspond to the task, as detailed below. Figure 5 As shown.

[0090] The loss mask is designed to only perform loss calculations on the portion corresponding to Epost, and other portions are not included in the loss calculation. Figure 4 As shown in (c).

[0091] (4) Epre and Epost training: Freeze the original weights of the large model, and only use Epre and Epost as trainable parameters; each training batch contains only samples of the same task type and shares a single attention mask. The rest of the training process is the same as that of a regular large model training, and the Epre and Epost weight parameters are saved after training.

[0092] 2. White-box distillation training: Optimizing the task compatibility between Epre and Epost. (1) Offline generation of original model logits: Input the training set of the target task into the complete large model (original model) for inference, obtain the logits of the entire input sequence, and save only the label sequence for subsequent dataset production.

[0093] In this embodiment, in order to further reduce the memory required to store logits, the logits corresponding to each token are calculated using the softmax function to obtain the probability of each word in the vocabulary, and only the 15 logits with the highest probability are stored.

[0094] (2) Distillation training dataset creation: The data construction method is the same as that in the fine-tuning training stage. The difference is that the ground truth value corresponding to the Epost in the output part is the logits generated by the original model. In this embodiment, except for the 15 logits stored for each token, all others are set to 0. The logits of other positions that are not Epost are set to any value (such as 0), because they do not participate in the loss calculation due to the setting of the lossmask.

[0095] (3) Epre and Epost training: The design of attention mask and loss mask is the same as the fine-tuning training above. The difference is that when calculating the loss, the positive KL divergence is used to evaluate the difference between the model output and the original model logits; iterative training is performed until the model loss converges. The final Epre and Epost weights are saved for subsequent inference acceleration.

[0096] IV. Accelerated Reasoning Stage: Adaptive Draft Generation and Verification Based on the selected draft layer and the trained Epre and Epost, inference acceleration is achieved through "draft generation - adaptive verification - dynamic length adjustment". The specific process is as follows: Figure 6 As shown: 1. Task adaptation initialization: Based on the current inference task type, load the corresponding Epost (length) for that task. With Epre (length) Weight.

[0097] 2. Draft layer inference and KV-Cache storage: After the input prompt is processed by the Embedding layer, it is concatenated with Epost and Epre to form the structure of "Epost + prompt embedding + Epre", which is then input into the large model and propagated to the draft layer k determined in step (1). Save the KV-Cache (key-value cache) corresponding to the Epost and prompt embeddings during the inference process for computation reuse in the subsequent verification stage; The output of the draft layer k is directly connected to the original model output layer to obtain the draft logits corresponding to Epre.

[0098] 3. Calculation of draft entropy: Calculate the entropy value of each logits in the draft (to measure the uncertainty of the draft).

[0099] in The token probability is the result of Softmax normalization of the draft logits.

[0100] 4. Adaptive draft validation: A manually set entropy threshold α (the smaller the entropy, the higher the certainty) is used to handle two cases: Directly accept drafts: After black-box distillation, the generated drafts already possess a certain level of confidence for the task. If the entropy value of all drafts is less than α (the confidence level meets the requirement), then the draft is directly used as the reasoning result, skipping subsequent verification and reducing computational load.

[0101] Further validation of the draft: If the entropy value of a draft is greater than α (insufficient confidence), then draft tokens are generated according to a preset sampling strategy (such as temperature sampling) and input into the complete large model for validation, specifically: (1) Before the draft layer k, reuse the "non-Epost KV-Cache" saved in step 2, and update the KV of the newly generated draft tokens to the cache; (2) After the draft layer k, continue to complete the calculation of the remaining Transformer layers and update the KV that is not in the cache to KV-Cache; (3) Verification logic: Based on the previous position token predicted by the original model, if the current draft token is consistent with the original model prediction, the draft is accepted; if the next draft token based on the current draft token is consistent with the original model prediction, it is accepted; if they are inconsistent, the draft is rejected and verification stops.

[0102] 5. Dynamic adjustment of Epre length: The entropy threshold β is manually set, and the entropy value of each draft token is calculated according to the order in which the drafts are generated: If the entropy of a draft token is greater than β (uncertainty increases), then the "total number of this token and all previous draft tokens" will be used as the Epre length of the task in the next inference. If the entropy of all draft tokens is less than β (uncertainty stable), then the next inference will still use the full length of the Epre from that task. ).

[0103] 6. Termination of reasoning: Repeat steps 2-5 until a complete inference result is generated (meeting the task output length requirement or generating a terminator), then end the inference process.

[0104] Tests showed that the draft method could accurately generate category names for SMS classification tasks, with an average speedup of 3.9 on 100 data points; an average speedup of 2.8 on 100 task intent recognition tasks; and an average speedup of 2.5 on 100 SMS summary tasks. Compared with existing methods, it achieved more significant speedups on specific tasks.

[0105] Figure 7 This is a schematic diagram of a device structure provided in an embodiment of the present invention. Figure 7 As shown, the device may include an acquisition module 710, a splicing module 720, an input module 730, and a reasoning module 740.

[0106] The acquisition module 710 is used to acquire the input sequence, the trained forward cueing embedding vector sequence, and the backward completion embedding vector sequence; The concatenation module 720 is used to concatenate the input embedding vector obtained by processing the input sequence through the embedding layer of the large language model with the forward cueing embedding vector sequence and the backward completion embedding vector sequence to obtain a combined input sequence. The input module 730 is used to input the combined input sequence into the large language model and propagate it forward to at least one draft generation layer to obtain candidate draft output information corresponding to the position of the forward cue embedding vector sequence; the draft generation layer is a Transformer layer selected from all Transformer layers of the large language model based on the dataset of the target reasoning task, which meets the reasoning accuracy condition and the reasoning speed condition. The inference module 740 is used to determine the inference result based on the confidence level of the candidate draft output information. The inference result is either the candidate draft output information or the result of submitting the token sequence sampled based on the candidate draft output information to all layers of the large language model for verification inference.

[0107] In one embodiment, the apparatus further includes: The selection module is used to select batches of target sample data from the dataset of the target inference task. The input module 730 is also used to input the target sample batch data into the large language model, sequentially obtain the output logic value of each Transformer layer, and record the inference time of each layer. The calculation module is used to calculate the precision difference between the output logic value of each layer and the output logic value of the final layer, and to calculate the inference speedup ratio of each layer relative to the final layer. The selection module is used to select the Transformer layer with the largest inference speedup from all Transformer layers whose accuracy difference is less than or equal to a preset accuracy loss threshold, and use it as the draft generation layer.

[0108] In one embodiment, the acquisition module 710 is specifically used for: Obtain the training dataset for the target reasoning task; Based on the training dataset of the target reasoning task, determine the initial length of the backward completion embedding vector sequence and the initial length of the forward cue embedding vector sequence; Initialize the parameters of the forward hint embedding vector sequence and the backward completion embedding vector sequence; The original parameters of the large language model are frozen. Using the training dataset, the combined input sequence is used as the model input to train the forward cue embedding vector sequence and the backward completion embedding vector sequence until the preset convergence condition is met, and the trained forward cue embedding vector sequence and backward completion embedding vector sequence are obtained.

[0109] In one embodiment, the acquisition module 710 is specifically used for: During the fine-tuning training phase, the real labels of the training dataset are used as supervision signals for training; During the white-box knowledge distillation training phase, the output logic values ​​generated by the large language model on the training dataset are used as supervision signals for training. During training, the forward cue embedding vector sequence uses a causal masking mode in the self-attention calculation; the input embedding vector and the backward completion embedding vector sequence use a bidirectional masking mode in the self-attention calculation.

[0110] In one embodiment, the inference module 740 is specifically used for: Based on the probability distribution of each token position in the candidate draft output information, calculate the entropy value used to quantify the confidence level; If the entropy value at all positions is lower than the preset first entropy threshold, then the candidate draft output information is determined as the reasoning result; If there is at least one position where the entropy value is higher than the first entropy threshold, the candidate draft output information is sampled to obtain a token sequence. The token sequence is submitted to all layers of the large language model for verification reasoning to obtain the reasoning result.

[0111] In one embodiment, the inference module 740 is specifically used for: Reuse the key-value cache that was previously computed and saved before the draft generation layer, corresponding to the input embedding vector and the backward completion embedding vector sequence; Update the key-value cache with the key-value sequence of tokens; Starting from the layer below the draft generation layer, the updated key-value cache is used as the input for the self-attention computation of the remaining Transformer layers to perform confirmatory reasoning and output the reasoning results.

[0112] In one embodiment, the inference module 740 is further configured to obtain feedback information of the candidate draft output information in the current round of inference, the feedback information including the entropy of the probability distribution of each token position in the candidate draft output information; The device also includes: The adjustment module is used to adjust the generation length of the forward cue embedding vector sequence during the next inference based on feedback information.

[0113] In one embodiment, the adjustment module is specifically used for: Based on the feedback information, the entropy of each token position in the candidate draft output is calculated sequentially in this round of inference; If the entropy of the nth token position exceeds the preset second entropy threshold, then n will be used as the length of the forward cue embedding vector sequence for the next inference. If the entropy of all token positions does not exceed the second entropy threshold, then the current length of the forward hint embedding vector sequence is maintained during the next inference.

[0114] The large language model inference method, apparatus, device, and computer storage medium of this invention acquire an input sequence, a trained forward cue embedding vector sequence, and a backward completion embedding vector sequence. The input embedding vector obtained by processing the input sequence through the embedding layer of the large language model is concatenated with the forward cue embedding vector sequence and the backward completion embedding vector sequence to obtain a combined input sequence. An enhanced context rich in task prior information is constructed at the model input end, injecting explicit generation guidance into the model without modifying the model's core parameters. The combined input sequence is input into the large language model and forward-propagated to at least one draft generation layer to obtain candidate draft output information corresponding to the position of the forward cue embedding vector sequence. The draft generation layer is a Transformer layer selected from all Transformer layers of the large language model based on the dataset of the target inference task, satisfying the inference accuracy and inference speed conditions. Through the draft generation layer, the model can output a high-quality, near-final intermediate representation before reaching the final layer, allowing most computations to terminate early at this intermediate layer. Based on the confidence level of the candidate draft output information, the inference result is determined. The inference result is either the candidate draft output information or the token sequence sampled from the candidate draft output information, which is submitted to all layers of the large language model for verification inference. This achieves adaptive acceleration of the inference process without requiring additional parallel model training or modification of the original model architecture. An evaluable fast generation path is established within the model, intelligently balancing inference efficiency and accuracy.

[0115] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention is shown.

[0116] An electronic device may include a processor 301 and a memory 302 storing computer program instructions.

[0117] Specifically, the processor 301 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0118] Memory 302 may include mass storage for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 302 may include removable or non-removable (or fixed) media, or memory 302 may be non-volatile solid-state memory. Memory 302 may be internal or external to the integrated gateway disaster recovery device.

[0119] In one instance, memory 302 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0120] Memory 302 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0121] The processor 301 reads and executes computer program instructions stored in the memory 302 to achieve... Figure 3 The large language model reasoning method in the illustrated embodiment.

[0122] Furthermore, in conjunction with the large language model reasoning method in the above embodiments, this invention can be implemented using a computer storage medium. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the large language model reasoning methods in the above embodiments.

[0123] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the large language model reasoning methods described in the above embodiments.

[0124] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0125] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0126] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0127] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0128] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A reasoning method for large language models, characterized in that, include: Obtain the input sequence, the trained forward cueing embedding vector sequence, and the backward completion embedding vector sequence; The input embedding vector obtained by processing the input sequence through the embedding layer of the large language model is concatenated with the forward cueing embedding vector sequence and the backward completion embedding vector sequence to obtain a combined input sequence. The combined input sequence is input into the large language model and forward propagated to at least one draft generation layer to obtain candidate draft output information corresponding to the position of the forward cue embedding vector sequence; The draft generation layer is a Transformer layer selected from all Transformer layers of a large language model based on a dataset of the target reasoning task, which meets the conditions for reasoning accuracy and reasoning speed. Based on the confidence level of the candidate draft output information, the inference result is determined. The inference result is either the candidate draft output information or the result of submitting the token sequence sampled based on the candidate draft output information to all layers of the large language model for verification inference.

2. The method according to claim 1, characterized in that, Before inputting the combined input sequence into the large language model and forward propagating it to at least one draft generation layer, the method further includes: Select target sample batch data from the dataset of the target inference task; The target sample batch data is input into the large language model, and the output logic value of each Transformer layer is obtained in sequence, and the inference time of each layer is recorded. Calculate the precision difference between the output logic value of each layer and the output logic value of the final layer, and calculate the inference speedup ratio of each layer relative to the final layer; From all Transformer layers whose accuracy difference is less than or equal to a preset accuracy loss threshold, select the Transformer layer with the largest inference speedup as the draft generation layer.

3. The method according to claim 1, characterized in that, The step of obtaining the trained forward cue embedding vector sequence and backward completion embedding vector sequence includes: Obtain the training dataset for the target inference task; Based on the training dataset of the target reasoning task, determine the initial length of the backward completion embedding vector sequence and the initial length of the forward cue embedding vector sequence; Initialize the parameters of the forward cue embedding vector sequence and the backward completion embedding vector sequence; The original parameters of the large language model are frozen. Using the training dataset, the combined input sequence is used as the model input to train the forward cue embedding vector sequence and the backward completion embedding vector sequence until the preset convergence condition is met, thus obtaining the trained forward cue embedding vector sequence and backward completion embedding vector sequence.

4. The method according to claim 3, characterized in that, The training of the forward cue embedding vector sequence and the backward completion embedding vector sequence includes: During the fine-tuning training phase, the real labels of the training dataset are used as supervision signals for training; During the white-box knowledge distillation training phase, the output logic values ​​generated by the large language model on the training dataset are used as supervision signals for training. In the training process, the forward cue embedding vector sequence employs a causal masking pattern in the self-attention calculation; the input embedding vector and the backward completion embedding vector sequence employ a bidirectional masking pattern in the self-attention calculation.

5. The method according to claim 1, characterized in that, The determination of the inference result based on the confidence level of the candidate draft output information includes: Based on the probability distribution of each token position in the candidate draft output information, calculate the entropy value used to quantify the confidence level; If the entropy value at all positions is lower than the preset first entropy threshold, then the candidate draft output information is determined as the reasoning result; If the entropy value at at least one position is higher than the first entropy threshold, the candidate draft output information is sampled to obtain a token sequence; The token sequence is submitted to all layers of the large language model for verification reasoning to obtain the reasoning result.

6. The method according to claim 5, characterized in that, The step of submitting the token sequence to all layers of the large language model for verification reasoning to obtain the reasoning result includes: The key-value cache corresponding to the input embedding vector and the backward completion embedding vector sequence, which was calculated and saved before the draft generation layer, is reused. Update the key value of the token sequence to the key value cache; Starting from the layer below the draft generation layer, the updated key-value cache is used as the input for the self-attention computation of the remaining Transformer layers to perform confirmatory reasoning and output the reasoning result.

7. The method according to claim 1, characterized in that, After determining the inference result based on the confidence level of the candidate draft output information, the method further includes: Obtain feedback information on the candidate draft output information in this round of reasoning, the feedback information including the entropy of the probability distribution of each token position in the candidate draft output information; Based on the feedback information, adjust the generation length of the forward cue embedding vector sequence for the next inference.

8. The method according to claim 7, characterized in that, The step of adjusting the generation length of the forward cue embedding vector sequence for the next inference based on the feedback information includes: Based on the feedback information, the entropy of each token position in the candidate draft output during this round of inference is calculated sequentially; If the entropy of the nth token position exceeds the preset second entropy threshold, then n will be used as the length of the forward hint embedding vector sequence for the next inference. If the entropy of all token positions does not exceed the second entropy threshold, then the current length of the forward hint embedding vector sequence is maintained during the next inference.

9. A large language model reasoning device, characterized in that, The device includes: The acquisition module is used to acquire the input sequence, the trained forward cueing embedding vector sequence, and the backward completion embedding vector sequence; The concatenation module is used to concatenate the input embedding vector obtained by processing the input sequence through the embedding layer of the large language model with the forward cueing embedding vector sequence and the backward completion embedding vector sequence to obtain a combined input sequence. An input module is used to input the combined input sequence into the large language model and propagate it forward to at least one draft generation layer to obtain candidate draft output information corresponding to the position of the forward cue embedding vector sequence; the draft generation layer is a Transformer layer selected from all Transformer layers of the large language model based on the dataset of the target inference task, which meets the inference accuracy condition and the inference speed condition. The inference module is used to determine the inference result based on the confidence level of the candidate draft output information. The inference result is the candidate draft output information, or the result of submitting the token sequence sampled based on the candidate draft output information to all layers of the large language model for verification inference.

10. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the large language model reasoning method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, implement the large language model reasoning method as described in any one of claims 1-8.

12. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the large language model reasoning method as described in any one of claims 1-8.

Images