A device-end large language model inference method, apparatus, device, and program product

Abstract

The application discloses a device-side large language model reasoning method and device, equipment and program product. The method comprises the following steps: inputting a reasoning task into a device-side large language model, and configuring a draft model set for the large language model, wherein the draft model set comprises draft models with different parameters; performing model routing selection processing on the draft model set through a pre-trained selector to obtain a selection model; performing generation probability calculation processing according to the word token confidence of the selection model, and determining the sequence length of the output of the selection model according to the calculated generation probability; and performing inference decoding processing on the large language model through the selection model according to the sequence length to obtain a reasoning result. The application can improve the reasoning efficiency of the device-side large language model, and can be widely applied in the field of artificial intelligence technology.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a device-side large language model reasoning method, apparatus, device, and program product. Background Technology

[0002] With increasing demands for privacy protection, personalization, and real-time services, deploying large language models (LLMs) directly on edge devices—a technique known as on-device LLM inference—has garnered significant attention. However, on-device LLM inference is constrained by substantial energy consumption and computational overhead, exacerbating inference latency and limiting real-time processing capabilities, thus impacting the efficiency of large language model inference on devices.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a device-side large language model reasoning method, apparatus, device, and program product that can improve the efficiency of device-side large language model reasoning.

[0005] To achieve the above objectives, one aspect of this application proposes a device-side large language model inference method, the method comprising: The inference task is input into a large language model deployed to the device, and a set of draft models is configured for the large language model, the set of draft models including multiple draft models with different parameters; The draft model set is processed by a pre-trained selector to obtain the selected model; The generation probability is calculated based on the word confidence of the selection model, and the length of the sequence output by the selection model is determined based on the calculated generation probability. Based on the sequence length, the large language model is subjected to inference decoding processing using the selection model to obtain the inference result.

[0006] In some embodiments, the process of performing model routing selection on the draft model set using a pre-trained selector to obtain a selected model includes the following steps: The selector is obtained by training the large language model. The inference task is feature-processed using the large language model to obtain task features; The selected model is obtained by selecting from the set of draft models based on the task characteristics using the selector.

[0007] In some embodiments, the process of training the selector using the large language model includes the following steps: Obtain training data; The training data is processed by feature extraction and concatenation using the large language model to obtain concatenated features. The confidence score of the spliced ​​features is obtained by performing confidence calculation on the selector. The reward value set is obtained by calculating the reward value of the draft model set based on the length of the accepted candidate sequence in the training data. Based on the context confidence score and the reward value set, the selector is processed by cross-entropy term and delay-aware regularization term loss calculation to obtain the loss value; The selector's parameters are adjusted based on the loss value to train the selector.

[0008] In some embodiments, the step of calculating the generation probability based on the confidence level of the selection model and determining the sequence length output by the selection model based on the calculated generation probability includes the following steps: The current lexical unit is determined based on the reasoning task, and the lexical confidence of the selection model is calculated based on the current lexical unit. The context confidence score of the selection model is output through the selector; The generation probability is obtained by calculating the generation probability based on the context confidence score and the word confidence score. Random values ​​are obtained by sampling random variables based on the current word; Based on the random value, the generation probability is processed using probabilistic soft decision-making to determine the sequence length output by the selection model.

[0009] In some embodiments, the step of performing probabilistic soft decision processing on the generation probability based on the random value to determine the sequence length output by the selection model includes the following steps: When the random value is less than the generation probability, a word element is generated through the selection model and the generated word element is used as the current word element. The process is then returned to the word element confidence score of the selection model calculated based on the current word element. When the random value is greater than or equal to the generation probability, stop generating words and determine the length of the sequence output by the selection model.

[0010] In some embodiments, the step of performing speculative decoding on the large language model based on the sequence length using the selection model to obtain the inference result includes the following steps: Candidate sequences are obtained by outputting the selection model based on the sequence length; The candidate sequences are input into the large language model for verification processing to obtain the verification results; Based on the verification results, the selection model is switched using the selector, and the process returns to the step of calculating the generation probability based on the word confidence of the selection model. The length of the sequence output by the selection model is determined based on the calculated generation probability, until the large language model completes the inference and obtains the inference result.

[0011] In some embodiments, the method further includes heterogeneous resource management on the device, comprising the following steps: The draft model set is reconstructed using a paginated key-value cache, and the reconstruction task is interleaved into pipeline gaps during the validation of the large language model. A hierarchical computational pipeline scheduling process is performed on the device, the draft model set, and the large language model to obtain a scheduling strategy. The resources on the device are scheduled according to the scheduling strategy.

[0012] To achieve the above objectives, another aspect of this application proposes a device-side large language model inference apparatus, the apparatus comprising: The model deployment module is used to deploy the inference task input to the large language model on the device and configure a draft model set for the large language model. The draft model set includes multiple draft models with different parameters. The model selection module is used to perform model routing selection on the draft model set through a pre-trained selector to obtain the selected model. The probability calculation module is used to perform generation probability calculation processing based on the word confidence of the selection model, and determine the sequence length output by the selection model based on the calculated generation probability. The inference decoding module is used to perform inference decoding on the large language model based on the sequence length and the selection model to obtain the inference result.

[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0016] The embodiments of this application include at least the following beneficial effects: This application provides a device-side large language model inference method, apparatus, device, and program product. This solution deploys the inference task input to a large language model on the device side and configures a draft model set for the large language model. It can predict inference lexical units through the draft model set, improving the inference efficiency of the large language model. Furthermore, this solution performs model routing selection processing on the draft model set through a pre-trained selector to obtain a selected model. It can dynamically select draft models with different parameter scales to dynamically adapt to the lexical distribution of the large language model during decoding, improving the decoding efficiency of the large language model. In addition, this solution also performs generation probability calculation processing based on the lexical confidence of the selected model and determines the sequence length output by the selected model based on the calculated generation probability. It can adjust the generation length of the draft model, thereby ensuring the generation of high-quality candidate sequences while reducing redundant computation and energy consumption. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the speculative decoding paradigm in related technologies; Figure 2 This is a schematic diagram illustrating the speculative draft hybrid paradigm used in an embodiment of this application; Figure 3 This is a schematic diagram of an implementation environment provided in the embodiments of this application; Figure 4 This is a flowchart of a device-side large language model inference method provided in an embodiment of this application; Figure 5 These are training example diagrams provided in the embodiments of this application; Figure 6 This is a flowchart of the sequence length adaptive output provided in the embodiments of this application; Figure 7 This is a schematic diagram of the scheduling strategy provided in an embodiment of this application; Figure 8 This is a schematic diagram of the CPU / GPU computing pipeline provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a device-side large language model inference device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0019] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0020] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0023] 1) Artificial Intelligence (AI) is a new technological science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems.

[0024] 2) Large Language Models (LLMs) are deep learning models trained on large amounts of text data, enabling them to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics by training on massive datasets. Their core idea is to learn patterns and structures of natural language through large-scale unsupervised training, thus mimicking the human language cognition and generation process to some extent.

[0025] 3) Mixture of Experts (MoE) is an advanced deep learning architecture that improves overall performance by integrating the predictions of multiple expert models. It involves dividing an artificial intelligence (AI) model into separate sub-networks (or "experts"), each specializing in a subset of the input data, to collectively perform a task.

[0026] 4) Speculative Decoding, also known as speculative decoding, is an innovative technique for accelerating reasoning in Large Language Models (LLMs). It significantly reduces inference time by using a smaller draft model to predict the next token and then using a larger target model to validate these predictions.

[0027] With increasing demands for privacy, personalization, and real-time services, deploying large language models (LLMs) directly on edge devices—a process known as on-device LLM inference—has garnered significant attention. This paradigm shift is crucial for human-computer dialogue systems and embodied intelligence. Despite its advantages, on-device LLM inference is limited by substantial energy consumption and computational overhead, exacerbating inference latency and restricting real-time processing capabilities. LLMs process user prompts as word-segmented sequences and progressively generate output tokens through autoregressive decoding. Each token is generated based on the prompt and previously generated tokens, without knowledge of future tokens. The computational complexity of this process leads to significant latency, which is further amplified by the limited processing power of edge devices. For example, deploying an LLaMA2-70B on an NVIDIA Jetson Orin using FP16 results in decoding speeds as low as 0.53 tokens / second while consuming approximately 63.6 joules / second. Therefore, optimizing decoding speed is critical for the feasibility of on-device LLM inference.

[0028] In related technologies, speculative decoding (SD) has emerged as a promising technique for accelerating large-scale language model (LLM) inference on-device platforms. For example... Figure 1As shown, SD (Simplified Derivatives) uses a small draft model to autoregressively predict multiple candidate tokens, which are then validated in parallel by a large target model. If the draft model successfully predicts a token acceptable to the target model, the overall inference latency on the device can be reduced. However, during decoding, the probability distribution of each token in the target model dynamically aligns with different draft models, meaning the draft model with the highest acceptance rate changes dynamically. In contrast, in the SD paradigm of related technologies, each token is sampled from the distribution of the same draft model, which cannot adapt to the dynamic changes in the target model's token distribution, resulting in suboptimal performance.

[0029] In view of this, this application provides a device-side large language model reasoning method, apparatus, device, and program product. This solution proposes a new speculative generation (SD) paradigm called "Mixture of Speculative Drafting" (MoSD). Figure 2 As shown, this paradigm selects draft models with different parameter scales to dynamically adapt to the token distribution of the target model during decoding. Furthermore, this application also proposes an efficient speculative decoding system for device-side Large Language Model (LLM) inference. This application employs a trainable, lightweight module that can be easily deployed on edge devices, real-time sensing of SD progress, and intelligently determining the draft model and its generation length. This application proposes an LLM routing mechanism that enables the target model to dynamically evaluate token prediction difficulty and switch between draft models at the token level, thereby improving the acceptance rate of candidate tokens. This application also introduces a confidence-based token adaptation strategy, adjusting the token generation length of the draft model based on the confidence scores of candidate tokens generated by the LLM routing and draft models. This method ensures high-quality candidate tokens while reducing redundant computation and energy consumption.

[0030] This application provides a device-side large language model inference method, relating to the field of artificial intelligence technology. This device-side large language model inference method can be applied to a terminal or to software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing device-side large language model inference, but is not limited to the above forms.

[0031] This application can be used in a wide variety of general-purpose or special-purpose computer device environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, microprocessor-based devices, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above devices, etc. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0032] Figure 3 This is a schematic diagram illustrating the implementation environment of a device-side large language model inference method provided in this application embodiment. (Refer to...) Figure 3 The main hardware and software components of this implementation environment include a terminal 301 and a server 302, which are communicatively connected. The method can be executed based on the interaction between the terminal 301 and the server 302. For example, the specific execution algorithm of the large language model inference method on the device side can be stored in the server 302. When the large language model is deployed to the terminal 301, the inference method is called to perform task inference processing on the large language model, obtaining the corresponding task inference result. Furthermore, the terminal 301 and the server 302 can be nodes in a blockchain; this embodiment does not specifically limit this.

[0033] Figure 4This is an optional flowchart of a device-side large language model inference method provided in an embodiment of this application. Figure 4 The method may include, but is not limited to, steps S401 to S404.

[0034] Step S401: Input the inference task into the large language model deployed to the device, and configure a draft model set for the large language model. The draft model set includes multiple draft models with different parameters. Step S402: Perform model routing selection processing on the draft model set through a pre-trained selector to obtain the selected model; Step S403: Calculate the generation probability based on the word confidence of the selection model, and determine the length of the sequence output by the selection model based on the calculated generation probability. Step S404: Based on the sequence length, perform speculative decoding processing on the large language model using the selection model to obtain the inference result.

[0035] Steps S401 to S404 of this embodiment involve deploying a large language model on a device to perform a corresponding inference task. The device can be a mobile phone, smartwatch, personal computer, in-vehicle terminal, or other terminal device. The inference task can be speech recognition, image recognition, text generation, language translation, or other tasks. It is conceivable that this inference task can include various input data, such as text data, speech data, and image data. This embodiment uses the large language model as the target model and configures a draft model set for it. The draft model set includes multiple draft models with different parameters. This embodiment can dynamically select draft models with different parameter scales, thereby dynamically adapting to the token distribution of the target model during decoding and improving the inference efficiency of the large language model. This embodiment also uses a pre-trained selector to perform model routing selection on the draft model set, selecting a suitable draft model as the selected model, and then using the selected model to perform inference decoding on the large language model. Furthermore, this embodiment also calculates the generation probability based on the lexical confidence of the selected model and adjusts the generation length of the selected model. The selected model outputs the corresponding candidate sequence based on the generated length and feeds it into the large language model for inference decoding to obtain the final inference result.

[0036] In step S401 of some embodiments, the inference task is input into a large language model deployed to the device, and a set of draft models is configured for the large language model, the set of draft models including multiple draft models with different parameters.

[0037] The reasoning task includes not only the original question or instruction input by the user, but also the contextual information provided by the task and specific reasoning requirements. Specifically, the reasoning task can include data processing tasks such as image recognition and speech recognition, and can include various types of data, such as text data and image data. The device can be a smartphone, tablet, embedded device, or edge computing node. The main language model is a deep learning model capable of reasoning from the user's input reasoning task to obtain the corresponding reasoning result, such as recognized text or text corresponding to speech data. The draft model set includes multiple draft models with different parameters. These draft models are small, efficient neural networks with a parameter size significantly smaller than the main main language model. These draft models are specifically trained to learn and simulate the output probability distribution of the main main language model in a given context, aiming to quickly generate candidate text with lower computational cost.

[0038] In this embodiment, user queries, contextual information, or inference requirements originating from cloud services or local applications can be encapsulated as structured data as inference tasks. These tasks are then transmitted to a large language model pre-optimized using model quantization and adaptation techniques and stored in the device's local memory via the inference runtime environment provided by the device's operating system. Simultaneously, a set of draft models, composed of multiple small, efficient neural networks, is loaded on the device. Each draft model in the set is trained to predict the subsequent output tokens of the large language model under specific token sequence conditions. This set of models collaborates with the large language model through a dynamic selection strategy. Based on the current input context, this strategy selects one or more of the most suitable draft models from the set, generates a candidate sequence in parallel, and submits this candidate sequence to the large language model for rapid verification and correction. This speculative decoding mechanism significantly improves the inference speed of the large language model by reducing its autoregressive decoding steps.

[0039] In step S402 of some embodiments, the draft model set is subjected to model routing selection processing through a pre-trained selector to obtain the selected model, including the following steps: The selector is obtained by training the large language model. The inference task is feature-processed using the large language model to obtain task features; The selected model is obtained by selecting from the set of draft models based on the task characteristics using the selector.

[0040] In this embodiment, a selector is trained using a large language model. This selector is used to fine-tune the lightweight LM head within the large language model to enhance selection decisions, thereby achieving feature-based dynamic draft model selection. This balances acceptance rate and latency, improving the inference efficiency of the large language model. In one feasible embodiment, a lightweight selector is trained internally within the large language model, constructed using three linear layers and a Tanh activation function. It is conceivable that this embodiment could also employ more complex variants, such as a shallow transformer as the selector. However, the selector constructed using three linear layers and a Tanh activation function in this embodiment achieves optimal performance with minimal overfitting.

[0041] It is important to note that in this embodiment, task features are obtained by performing feature processing on the reasoning task using a large language model. These task features can be obtained through the output of the transformer layer of the large language model. Then, a selector selects a suitable draft model from the draft model set based on the task features. Through this selector, this embodiment can dynamically evaluate the token prediction difficulty and switch between draft models at the token level, thereby improving the acceptance rate of candidate tokens.

[0042] In some embodiments, the process of training the selector using the large language model includes the following steps: Obtain training data; The training data is processed by feature extraction and concatenation using the large language model to obtain concatenated features. The confidence score of the spliced ​​features is obtained by performing confidence calculation on the selector. The reward value set is obtained by calculating the reward value of the draft model set based on the length of the accepted candidate sequence in the training data. Based on the context confidence score and the reward value set, the selector is processed by cross-entropy term and delay-aware regularization term loss calculation to obtain the loss value; The selector's parameters are adjusted based on the loss value to train the selector.

[0043] In this embodiment, a selector is trained using a large language model. Specifically, the large language model extracts and concatenates features from the training data to obtain concatenated features. The training data can be text data, image data, etc., required for image recognition, text recognition, etc. In this embodiment, the large language model outputs token features from the training data, concatenates these token features, and then inputs the concatenated features into the selector for confidence calculation. The confidence calculation formula is as follows: =Tanh( 1 2) 3, =[ 1, , D ]; in, This represents the calculated context confidence score, which can be used to identify the draft model set [ 1, , D The draft model is selected as the model to be selected. Indicates splicing characteristics, 1. 2 and 3 represents the weight parameters of the three linear layers. This embodiment captures the relationship between the draft model and the token through compact splicing features. Then, this embodiment further calculates the reward value of the draft model set based on the accepted length of the candidate sequences in the training data, obtaining a reward value set. The candidate sequences are generated by predicting from each draft model in the draft model set based on the training data.

[0044] With a set of heterogeneous draft models For example, where j∈ ={1,2, , }, This represents the number of draft models, with the large language model set as the target model. And the training data is set as .make T represents the i-th token. a：b : Represents a sequence { a , , b In the autoregressive decoding process, from the draft model set... ( 1:i Extract the token predicted by the draft model j from the data. i+1 , recorded as That is, the accuracy of the token predicted by the draft model j is .

[0045] This application's embodiment defines the reward for the draft model j. j Based on the candidate sequence length of The number of adopted tokens is The formula for calculating the reward value for j is as follows: ; in, j is the model-specific penalty (slower models have smaller penalties). Accepted tokens increase rewards, while discarded tokens decrease rewards.

[0046] This application embodiment can also convert the reward into a probability distribution, wherein the calculation formula for the probability distribution is as follows: ; in, This represents the calculated probability; Temperature is used to control smoothness. In this embodiment, the selector is processed by calculating the cross-entropy term and the delay-aware regularization term loss based on the context confidence score and the reward value set to obtain the loss value. The formula for calculating the loss value is as follows: ; In the formula, Indicates the loss value; Indicates the output of the selector; This represents the normalized weight for the accepted token. = 1; For the first rejected token, Other situations = 0. In the formula for calculating this loss value, the first term is used to encourage matching with the target distribution; the second term is used to promote efficient model selection and minimize decoding latency. Embodiments of this application can adjust the parameters of the selector using the calculated loss value, thereby obtaining a trained selector.

[0047] The embodiments of this application can select dynamic draft models based on features through a selector, which can achieve a balance between acceptance rate and latency. It can be easily deployed on edge devices, perceive SD progress in real time, and intelligently determine the draft model and its generation length, thereby improving the inference efficiency of large language models.

[0048] In step S403 of some embodiments, the step of calculating the generation probability based on the confidence level of the selection model and determining the sequence length output by the selection model based on the calculated generation probability includes the following steps: The current lexical unit is determined based on the reasoning task, and the lexical confidence of the selection model is calculated based on the current lexical unit. The context confidence score of the selection model is output through the selector; The generation probability is obtained by calculating the generation probability based on the context confidence score and the word confidence score. Random values ​​are obtained by sampling random variables based on the current word; Based on the random value, the generation probability is processed using probabilistic soft decision-making to determine the sequence length output by the selection model.

[0049] In the embodiments of this application, please refer to Figure 5 This application proposes a confidence-based adaptive segmentation length mechanism to dynamically determine the candidate segmentation length of each draft model, thereby reducing computational and energy costs. Unlike previous methods that use a fixed threshold to determine the confidence of the draft model, this application simultaneously utilizes the selector output of the target model and the information from the draft model to calculate the predicted length.

[0050] For example, in this application embodiment, a token to be predicted can be obtained based on the reasoning task, and the token to be predicted is used as the current word unit. This is achieved using a set of heterogeneous draft models. For example, in the autoregressive decoding process, from the draft model set ( 1:i Extract the token predicted by the draft model j from the data. i+1 , recorded as That is, the accuracy of the tokens predicted by the draft model j is In this embodiment, the accuracy is used as the token confidence score of the selection model. The context confidence score of the selection model is output by the selector, which is obtained by concatenating features into the selector output. Since validation rejects all tokens after the first failure, accurate length prediction is crucial. In each generation step, the selector outputs the context confidence score of the selected draft model. This score implicitly reflects the reward for longer sequences shaped by delay regularization training. Embodiments of this application will... Token-level confidence compared to the draft model Combined, to calculate the first The probability of generating another token is calculated using the following formula: ; In the formula, λ is used to balance the confidence levels of the draft model and the target model.

[0051] The embodiments of this application employ a confidence-based token adaptive strategy to adjust the candidate length using the confidence of the selector and the drafting model, thereby reducing computation and improving decoding efficiency.

[0052] In some embodiments, the step of performing probabilistic soft decision processing on the generation probability based on the random value to determine the sequence length output by the selection model includes the following steps: When the random value is less than the generation probability, a word element is generated through the selection model and the generated word element is used as the current word element. The process is then returned to the word element confidence score of the selection model calculated based on the current word element. When the random value is greater than or equal to the generation probability, stop generating words and determine the length of the sequence output by the selection model.

[0053] In this embodiment of the application, for each lexical unit, a random variable γ is sampled. ([0,1]). If γ < (cont.|~ If the candidate sequence is selected, generation continues; otherwise, lexical generation stops, and the candidate sequence is passed to the target model. Figure 6 As shown, this probabilistic soft decision-making strategy allows for the generation of longer sequences in contexts with high confidence, while generating shorter sequences in contexts with high uncertainty. The length of the sequence output by the selection model can be determined based on the length of the candidate sequences generated by the selection model.

[0054] In step S404 of some embodiments, the step of performing speculative decoding on the large language model based on the sequence length using the selection model to obtain the inference result includes the following steps: Candidate sequences are obtained by outputting the selection model based on the sequence length; The candidate sequences are input into the large language model for verification processing to obtain the verification results; Based on the verification results, the selection model is switched using the selector, and the process returns to the step of calculating the generation probability based on the word confidence of the selection model. The length of the sequence output by the selection model is determined based on the calculated generation probability, until the large language model completes the inference and obtains the inference result.

[0055] In this embodiment, the calculated sequence length allows control over the selection model's output to obtain candidate sequences of corresponding lengths, thereby adjusting the token generation length of the draft model. The candidate sequences are then input into the large language model for verification, yielding verification results. This embodiment can reselect the draft model based on the verification results. For example, if some tokens fail the verification, the draft model can be reselected using those tokens, switching the selected model and returning to steps such as calculating generation probability based on the word confidence of the selected model, until the large language model completes inference, i.e., all verification results pass, resulting in the inference result of the large language model, thus improving the model's inference capability.

[0056] It is conceivable that, in an inference task, multiple draft models can be dynamically selected by a selector for token prediction, and the length of the predicted token needs to be controlled by the selector. For example, in an inference task, multiple draft models can be selected simultaneously as selection models by a selector, and generation probability calculations can be performed for multiple selection models, thereby assigning corresponding sequence lengths to multiple selection models. The sequence length controls the output prediction sequence of the corresponding selection model to the large language model for verification.

[0057] This application's embodiments introduce a confidence-based token adaptation strategy, which adjusts the token generation length of the draft model based on the confidence scores of candidate tokens generated by the LLM routing and draft model. This method reduces redundant computation and energy consumption while ensuring high-quality candidate tokens.

[0058] In some embodiments, the method further includes heterogeneous resource management on the device, comprising the following steps: The draft model set is reconstructed using a paginated key-value cache, and the reconstruction task is interleaved into pipeline gaps during the validation of the large language model. A hierarchical computational pipeline scheduling process is performed on the device, the draft model set, and the large language model to obtain a scheduling strategy. The resources on the device are scheduled according to the scheduling strategy.

[0059] In this embodiment, the inference method further includes heterogeneous resource management on the device side. Related research has explored SD pipeline scheduling, primarily through GPU-only and CPU / GPU model-level strategies. However, both methods are not ideal for edge devices due to low computational efficiency or I / O overhead. This embodiment addresses these limitations by first introducing a paging key-value (KV) cache reconstruction strategy and then proposing a CPU / GPU parallel scheduling algorithm optimized for MoSD. As shown in Figure 7, the method in this embodiment effectively reduces pipeline stalls and improves resource utilization of edge hardware.

[0060] Specifically, in this embodiment, switching draft models requires rebuilding the KV cache because the cache is model-specific. This introduces latency because rebuilding involves forward propagation of the target draft model using newly generated tokens. As shown in Figure 7, once the selector selects draft model 2, the KV cache rebuild must be completed before drafting to preserve context. By mapping the attention layer to the CPU and the FFN layer to the GPU, the memory locality of the CPU and the computational power of the GPU are utilized. However, this layer-by-layer partitioning introduces idle time due to data dependencies—for example, the GPU needs to wait for the CPU to complete attention computation during target verification. To mitigate idle time, this embodiment utilizes the independence of KV cache rebuilding and target model verification. However, simply running the rebuild ( This can still interfere with SD execution. To address this, this application proposes a paginated KV reconstruction scheme by dividing KV tasks into blocks and interleaving them into pipeline gaps during target verification. Given an activated draft model j and its candidate tokens, this application proactively reconstructs the KV cache (by layer) for all other draft models to accommodate the gap size. Once verification is complete, the cache of rejected tokens is discarded. This method reduces idle time and improves ASTRA pipeline efficiency. Figure 7 As shown, the size of the blocks varies with workload and operating equipment. Draft models 1 and 2 have layers 1 and 2, respectively, while the target model has 3 layers.

[0061] Furthermore, this application also proposes a fine-grained GPU-CPU scheduling strategy to optimize the validation of the target model and the reconstruction of the key-value cache by interleaving tasks and reducing pipeline idle time. The main challenge lies in scheduling key-value reconstruction across draft models without interfering with validation. This application modeles this as a 0-1 integer programming problem. Given hardware... Draft model and have The target model of a Transformer block The objective of this application's embodiments is to find an optimal schedule. This is to maximize utilization during verification idle periods. The embodiments of this application first address... Perform offline analysis: , These represent the CPU / GPU computation time for the nth layer of the draft model, respectively. , for The computation time. Let ∈ {0,1} indicates whether the attention layer of the draft model n is in The m-th FFN (GPU) stage runs. exist The m-th attention (CPU) stage has an FFN layer. The corresponding 0-1 integer programming problem is defined as shown in Algorithm 1, and the expression of Algorithm 1 is as follows: ; The embodiments of this application can be solved using solvers such as Gurobi or CPLEX. Figure 8 The diagram illustrates a computational pipeline that interleaves execution of a target model and multiple draft models on a CPU and GPU. First, pointers to the attention layer and feedforward layer are initialized for each draft model. Then, each layer m of the target model is computed sequentially on both the CPU and GPU. This is based on the control matrix. and Whether it is 1 determines whether to execute the layer currently pointed to by the draft model in the corresponding CPU / GPU stage and increment the pointer, thereby overlapping the use of hardware computing power in the same layer.

[0062] This application's embodiments address these limitations by first introducing a paging KV cache reconstruction strategy and then proposing a CPU / GPU parallel scheduling algorithm optimized for MoSD, which effectively reduces pipeline stalls and improves the resource utilization of edge hardware.

[0063] The solutions of this application embodiment will be described in detail and explained below with reference to specific application examples: This application embodiment can be applied to the field of artificial intelligence technology, particularly to application scenarios such as speech recognition, text recognition, and image processing. Taking image recognition as an example, this application embodiment inputs the image to be recognized, along with corresponding contextual information and prompts, into a large language model as the inference task. This large language model has been deployed on a terminal device and configured with multiple draft models. This application embodiment uses a pre-trained selector to perform model routing selection on the draft model set, which can extract features from the data in the inference task. The selector then processes the extracted features to select the chosen model. This application embodiment also uses the selector to predict the output of the chosen model and determine the output length of the chosen model. Finally, based on the specified generation length, the selected model outputs the corresponding predicted sequence to the large language model for verification, obtaining the corresponding image recognition result. This approach ensures the generation of high-quality candidate sequences while reducing redundant computation and energy consumption, thus improving the inference efficiency of the large language model on the device.

[0064] Please see Figure 9 This application also provides a device-side large language model inference apparatus, which can implement the above-described device-side large language model inference method. The apparatus includes: The model deployment module 901 is used to deploy the inference task input to the large language model on the device and configure a draft model set for the large language model, wherein the draft model set includes multiple draft models with different parameters. The model selection module 902 is used to perform model routing selection on the draft model set through a pre-trained selector to obtain the selected model; The probability calculation module 903 is used to perform generation probability calculation processing based on the word confidence of the selection model, and determine the sequence length output by the selection model based on the calculated generation probability. The inference decoding module 904 is used to perform inference decoding processing on the large language model based on the sequence length through the selection model to obtain the inference result.

[0065] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0066] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0067] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0068] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store operating devices and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0069] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0070] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0071] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0072] This application provides a device-side large language model inference method, apparatus, device, and program product. This solution deploys the inference task to a large language model on the device and configures a draft model set for the large language model. It can predict inference lexical units through the draft model set, improving the inference efficiency of the large language model. Furthermore, this solution uses a pre-trained selector to perform model routing selection on the draft model set, obtaining a selected model. It can dynamically select draft models with different parameter scales to dynamically adapt to the lexical distribution of the large language model during decoding, improving the decoding efficiency of the large language model. In addition, this solution calculates the generation probability based on the lexical confidence of the selected model and determines the sequence length output by the selected model based on the calculated generation probability. It can adjust the generation length of the draft model, thereby ensuring the generation of high-quality candidate sequences while reducing redundant computation and energy consumption.

[0073] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0074] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0075] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Those skilled in the art will understand that all or some of the steps, apparatuses, or functional modules / units in the methods disclosed above can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0077] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0078] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between the devices or units may be through some interfaces, and the indirect coupling or communication connection may be electrical, mechanical, or other forms.

[0080] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0082] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A device-side large language model inference method, characterized in that, The method includes the following steps: The inference task is input into a large language model deployed to the device, and a set of draft models is configured for the large language model, the set of draft models including multiple draft models with different parameters; The draft model set is processed by a pre-trained selector to obtain the selected model; The generation probability is calculated based on the word confidence of the selection model, and the length of the sequence output by the selection model is determined based on the calculated generation probability. Based on the sequence length, the large language model is subjected to inference decoding processing using the selection model to obtain the inference result; The process of performing model routing selection on the draft model set using a pre-trained selector to obtain the selected model includes the following steps: The selector is obtained by training the large language model. The inference task is feature-processed using the large language model to obtain task features; The selected model is obtained by selecting from the draft model set according to the task characteristics using the selector; The process of training the selector using the large language model includes the following steps: Obtain training data; The training data is processed by feature extraction and concatenation using the large language model to obtain concatenated features. The confidence score of the spliced ​​features is obtained by performing confidence calculation on the selector. The reward value set is obtained by calculating the reward value of the draft model set based on the length of the accepted candidate sequence in the training data. Based on the context confidence score and the reward value set, the selector is processed by cross-entropy term and delay-aware regularization term loss calculation to obtain the loss value; The selector's parameters are adjusted based on the loss value to train the selector. The process of calculating the generation probability based on the confidence level of the selection model, and determining the sequence length output by the selection model based on the calculated generation probability, includes the following steps: The current lexical unit is determined based on the reasoning task, and the lexical confidence of the selection model is calculated based on the current lexical unit. The context confidence score of the selection model is output through the selector; The generation probability is obtained by calculating the generation probability based on the context confidence score and the word confidence score. Random values ​​are obtained by sampling random variables based on the current word; Based on the random value, the generation probability is processed using probabilistic soft decision-making to determine the sequence length output by the selection model.

2. The method according to claim 1, characterized in that, The step of performing probabilistic soft decision processing on the generated probabilities based on the random values ​​to determine the sequence length output by the selection model includes the following steps: When the random value is less than the generation probability, a word element is generated through the selection model and the generated word element is used as the current word element. The process is then returned to the word element confidence score of the selection model calculated based on the current word element. When the random value is greater than or equal to the generation probability, stop generating words and determine the length of the sequence output by the selection model.

3. The method according to claim 1, characterized in that, The step of performing inference decoding on the large language model based on the sequence length using the selection model to obtain the inference result includes the following steps: Candidate sequences are obtained by outputting the selection model based on the sequence length; The candidate sequences are input into the large language model for verification processing to obtain the verification results; Based on the verification results, the selection model is switched using the selector, and the process returns to the step of calculating the generation probability based on the word confidence of the selection model. The length of the sequence output by the selection model is determined based on the calculated generation probability, until the large language model completes the inference and obtains the inference result.

4. The method according to any one of claims 1 to 3, characterized in that, The method also includes heterogeneous resource management on the device side, including the following steps: The draft model set is reconstructed using a paginated key-value cache, and the reconstruction task is interleaved into pipeline gaps during the validation of the large language model. A hierarchical computational pipeline scheduling process is performed on the device, the draft model set, and the large language model to obtain a scheduling strategy. The resources on the device are scheduled according to the scheduling strategy.

5. A device-side large language model inference apparatus, characterized in that, The device is applied to a device-side large language model inference method as described in claim 1, and the device comprises: The model deployment module is used to deploy the inference task input to the large language model on the device and configure a draft model set for the large language model. The draft model set includes multiple draft models with different parameters. The model selection module is used to perform model routing selection on the draft model set through a pre-trained selector to obtain the selected model. The probability calculation module is used to perform generation probability calculation processing based on the word confidence of the selection model, and determine the sequence length output by the selection model based on the calculated generation probability. The inference decoding module is used to perform inference decoding on the large language model based on the sequence length and the selection model to obtain the inference result.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 4.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images