Distributed speculation decoding system for edge device

Abstract

The invention discloses a distributed speculation decoding system for edge equipment, and belongs to the technical field of large language model reasoning. The invention aims to solve the problems of high reasoning delay and load imbalance caused by static division and a traditional speculation scheme when a large language model is deployed on heterogeneous edge equipment in the prior art. The system comprises an off-line preprocessing module which is used for generating a model layer division strategy and an initial entropy triggering threshold value based on equipment resources; the online reasoning module is used for dynamically predicting the entropy of the middle layer of the model during decoding and determining a speculation triggering layer based on an entropy triggering threshold value dynamically adjusted according to the real-time equipment utilization rate; and transmitting a draft token on the trigger layer, and transmitting the draft token to downstream equipment for parallel verification. According to the method, the reasoning delay is effectively reduced, the utilization rate of heterogeneous equipment is improved, and the self-adaption to the dynamic load is realized.

Description

Technical Field

[0001] This invention relates to the field of large language model inference technology, and more particularly to a distributed speculative decoding system for edge devices. Background Technology

[0002] With the widespread application of large language models in natural language processing, code generation, and decision support, deploying them on edge devices closer to the data source to meet the needs of low latency, high privacy, and weak network connectivity scenarios (such as autonomous driving, industrial control, and augmented reality) has become an important industry trend.

[0003] Traditional centralized cloud deployments, while offering abundant computing resources, face high privacy risks due to data backhauling to the cloud and heavily rely on stable, high-bandwidth network connections, making them unsuitable for real-time interaction. Edge inference, by deploying models or some of their computational tasks locally or on collaborative edge devices, effectively reduces latency and protects data privacy.

[0004] However, deploying large language models in heterogeneous edge environments (i.e., environments with vastly different computing power, memory, and energy consumption) faces significant challenges. Existing distributed inference technologies, such as Tensor Parallelism (TP), incur huge communication overhead in low-bandwidth edge scenarios. While Pipeline Parallelism (PP) can reduce communication, traditional static layer partitioning strategies cannot adapt to the dynamically changing loads of edge devices (such as temporary task preemption and device offline), easily leading to load imbalance, where some high-performance devices are idle while low-performance devices become bottlenecks. Simultaneously, speculative decoding techniques designed to accelerate LLM autoregressive decoding also have adaptability issues in edge environments. Existing solutions either rely on additional draft models, which are difficult to handle on edge devices where memory is already strained (often exceeding the 80% threshold); or they only utilize features from the model's terminal layer for inference, missing the opportunity to use features from the model's intermediate layers for early prediction and failing to fully utilize the distributed resources of multiple devices.

[0005] Therefore, a pressing technical problem to be solved in this field is how to achieve efficient, low-latency inference of large language models on heterogeneous edge devices where computing, memory, and bandwidth resources are limited. Existing technologies have failed to provide a mechanism that can collaboratively optimize model partitioning and speculative decoding, resulting in the system's inability to simultaneously achieve low inference latency and high device utilization in dynamically changing edge environments, leading to load imbalance and resource waste. Summary of the Invention

[0006] The main objective of this invention is to provide a distributed speculative decoding system for edge devices to solve the problems raised in related technologies.

[0007] To achieve the above objectives, according to one aspect of the present invention, a distributed speculative decoding system for edge devices is provided, comprising: The offline preprocessing module is used for: Based on the resource information of the edge device cluster and the model characteristics of the target large language model, a layer partitioning strategy for the model on the edge devices is generated; and Based on historical decoding trajectories, initial speculative decoding parameters are determined, wherein the initial speculative decoding parameters include at least an initial entropy trigger threshold; The online inference module, connected to the offline preprocessing module, is used for: In the decoding step, inference is performed on the edge device according to the layer partitioning strategy; During the inference process, the prediction distribution entropy of the intermediate layer of the model is dynamically predicted; Based on the predicted entropy and a dynamically adjusted entropy trigger threshold, a speculative triggering layer for launching drafts is determined. When the inference reaches the speculation trigger layer, at least one draft token is generated; and On the downstream edge device following the speculative triggering layer, the at least one draft token is verified in parallel to obtain the final decoded token.

[0008] Furthermore, the online inference module is also used for: Real-time monitoring of the device utilization rate of the edge devices; and The entropy trigger threshold is dynamically adjusted based on the deviation between the real-time device utilization rate and a reference utilization rate.

[0009] Furthermore, the online inference module dynamically adjusts the entropy trigger threshold in the following manner: When the real-time utilization rate is higher than the reference utilization rate, the entropy trigger threshold is increased to trigger speculation at a layer with lower entropy. When the real-time utilization rate is lower than the reference utilization rate, the entropy trigger threshold is reduced to trigger speculation at a higher entropy layer.

[0010] Furthermore, the online inference module predicts the predicted distribution entropy of the intermediate layer using a Kalman filter.

[0011] Furthermore, the offline preprocessing module is also used for: The layer partitioning strategy, the initial entropy trigger threshold, and the number of candidate drafts are jointly optimized, with the optimization objective being to minimize the estimated end-to-end latency and device load imbalance.

[0012] Furthermore, the online inference module is also used in the parallel verification for: Obtain the final decoded token calculated from the main path; Compare whether the final decoded token is in the set of at least one draft token; If so, it is determined to be a speculative hit, and the parallel computation result corresponding to the draft token is accepted; If not, it is determined to be a speculative mismatch, and the result is rolled back to the main path calculation result.

[0013] Furthermore, the online inference module is also used for: When the speculative mismatch occurs, the entropy trigger threshold is adjusted in subsequent decoding steps to postpone speculative triggering. When the speculative hit occurs, the entropy trigger threshold is adjusted in subsequent decoding steps to trigger the speculative hit earlier.

[0014] Furthermore, when determining the initial speculative decoding parameters, the offline preprocessing module is also used to: By fitting a Sigmoid model, the relationship between draft hit rate and entropy and layer depth under different layers and different numbers of candidate drafts is modeled.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention achieves deep collaboration between layered speculative decoding and heterogeneous device resources by jointly optimizing model layer partitioning and initial speculative strategies in the offline phase and introducing a dynamic triggering mechanism based on entropy prediction in the online phase. Compared with existing technologies, this invention can dynamically initiate speculation in advance in the low-entropy region of the model's intermediate layer (rather than only the terminal layer) during inference, and hand over the draft token to downstream devices for parallel verification. This cross-device "parallel speculation + verification" mechanism effectively overlaps the time of computation, communication, and entropy prediction, significantly reducing idle "bubbles" in traditional pipeline parallelism, thereby significantly reducing end-to-end inference latency while ensuring that the generated results are completely consistent with the standard decoding.

[0016] Furthermore, this invention introduces a feedback adjustment mechanism based on real-time device utilization. The system can sense changes in the load of edge devices online (such as CPU / GPU utilization and task queues) and adaptively adjust the entropy trigger threshold accordingly, thereby dynamically controlling the aggressiveness of the speculative strategy (such as advancing or delaying the speculative trigger layer). This dynamic adaptability allows the system to automatically adopt a more conservative strategy to avoid frequent rollbacks when the load on some devices increases; and to revert to an aggressive strategy to maximize parallel gains when the load decreases. This effectively solves the load imbalance problem in heterogeneous edge environments and significantly improves the average utilization of edge devices. Attached Figure Description

[0017] Figure 1This is a schematic diagram comparing the inference time of the EdgeSpec scheme and the traditional pipeline parallel (PP) scheme in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall framework of the distributed speculative decoding system provided in an embodiment of the present invention, illustrating the offline preprocessing stage and the online inference stage; Figure 3 This is a schematic diagram illustrating the effect of the online adaptive adjustment strategy in an embodiment of the present invention, showing the relationship between the trigger layer, device utilization and decoding time, and entropy as a function of the number of layers; Figure 4 This is a schematic diagram comparing the effects of different entropy thresholds on system equipment utilization, inference latency, and speculative hit rate in embodiments of the present invention.

[0018] Figure labeling: 100, Offline preprocessing module; 110, Resource awareness and model partitioning module; 120, Speculative strategy modeling module; 130, Joint strategy search module; 200, Online inference module; 210, Dynamic decision-making module; 220, Hierarchical speculation module; 230, Parallel verification module; 240, Verification and rollback module; 250, Feedback update module. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this embodiment can be combined with each other. The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] This invention provides a distributed speculative decoding system for edge devices, aiming to solve the problems of load imbalance, high inference latency, and poor dynamic adaptability caused by heterogeneous device resources, limited network bandwidth, and the serial characteristics of autoregressive decoding when large language models are deployed on heterogeneous edge devices.

[0021] like Figure 2 As shown, the system framework proposed in this embodiment of the invention mainly includes two stages: an offline preprocessing stage and an online inference stage. The system is physically deployed on a platform consisting of... heterogeneous edge devices In a collaborative reasoning environment, each device... Having different computing capabilities Video memory capacity Inter-device bandwidth .

[0022] The system can be logically divided into the following functional modules: Offline preprocessing module 100: responsible for generating the optimal model partitioning and initializing the speculative strategy based on device resources and model characteristics before the inference task begins.

[0023] Online Inference Module 200: Responsible for receiving user requests at runtime, performing efficient speculative decoding, and dynamically adjusting strategies based on real-time status to minimize latency and ensure load balancing.

[0024] The following section provides a detailed explanation of each module and its collaborative working methods.

[0025] like Figure 2 As shown on the left, the offline preprocessing module 100 is mainly responsible for policy initialization, which can be further divided into the following sub-modules: 1. Resource Awareness and Model Partitioning Module 110 This module first collects all Resource information for each edge device, including computing power. Video memory capacity and bandwidth .

[0026] Subsequently, the module addresses a containing The Transformer model with each decoding layer is partitioned using pipeline parallelism (PP). This partitions the model into... A continuous layer, each device Allocate a continuous layer .

[0027] This module will enumerate all legal model partitioning schemes that satisfy the memory constraints. The constraints are: ; in, It is assigned to the device The number of floors, This refers to the memory usage of parameters in a single-layer Transformer. It is the maximum allocatable video memory ratio (e.g.) (This is to reserve space for dynamic operation).

[0028] 2. Speculative Strategy Modeling Module 120 This module models key parameters of speculative decoding by analyzing a large amount of historical decoding trajectory data: Top-k hit rate modeling: This module statistically analyzes the top-k hit rate. Layers use top- Draft hit rate of the strategy And fit the model using the Sigmoid model: ; in, For the Sigmoid function, This represents the total number of layers in the model. For the first The entropy of the layer prediction distribution. These parameters were obtained through offline trajectory fitting. The model indicates that the deeper the layer (…), the better. (smaller) or entropy The higher the value, the lower the hit rate.

[0029] 3. Joint Strategy Search Module 130 The goal of this module is to find an optimal set of initial parameters. That is, the optimal model partitioning Initial entropy trigger threshold and the number of candidate drafts To minimize a comprehensive optimization objective function : ; in: It is a given partition The Prefill phase is delayed.

[0030] It is the first Expected decoding latency for each token.

[0031] It represents the maximum imbalance between devices, used to constrain load balancing. This is a weighting factor.

[0032] This is the total number of tokens generated.

[0033] This module satisfies the constraints by enumerating them. Combination, selection The minimized solution serves as the initial configuration for the online inference phase.

[0034] like Figure 2 As shown on the right, the online inference module 200 is responsible for executing the actual decoding process. Its core idea is "layered speculative decoding," which means generating draft tokens in advance in the middle layer of the model and using downstream devices for parallel verification, thereby overlapping computation and communication and reducing waiting bubbles.

[0035] 1. Dynamic Decision Module 210 This module is used in each decoding step. The dynamics that determine speculative strategies are the "brain" of the system.

[0036] Entropy prediction: To determine in advance which layers of the model have high confidence (low entropy), this module uses a Kalman filter to predict the entropy in the sequence. Layer entropy .

[0037] State vector ,in It is normalized entropy , It refers to the size of the vocabulary. It is the first derivative of entropy with respect to the layer.

[0038] State transition matrix Assume that the rate of entropy change is approximately constant.

[0039] Observation matrix That is, directly observed entropy .

[0040] Process noise and observation noise This was obtained through offline data statistics.

[0041] Through Kalman filtering, the system can achieve the following: Predicting before the layer calculation is completed. The entropy of the layer shows a decreasing trend.

[0042] Dynamic threshold adjustment: This module adjusts the threshold based on real-time device utilization. Dynamically update entropy trigger threshold : ; in, It is an initial threshold determined offline. It is the reference utilization rate, and EMA is the moving average. It adjusts the speed coefficient. When the equipment load is high ( When ), increase the threshold. This makes speculation more conservative (requiring lower entropy to trigger) and reduces ineffective rollbacks; conversely, it makes speculation more aggressive.

[0043] Trigger layer selection: Combining entropy prediction and dynamic threshold, this module selects the current step. The earliest triggering layer : ; in It is the predicted number Layer entropy.

[0044] 2. Layered Speculation Module 220 (Draft Generation) When the inference pipeline executes to the trigger layer determined by the dynamic decision module 210 Time (e.g.) Figure 2 (As shown in "Decode1" and "DraftLayer"), this module is activated.

[0045] It obtains the first Hidden state of a layer And project it onto the vocabulary space: ; ; in It outputs the projection matrix. This module is based on... The probability distribution, select top- The most likely token As a draft token.

[0046] 3. Parallel Validation Module 230 (Parallel Prediction) This module is responsible for verifying the validity of the draft token. It will draft token Forming a large-scale speculation .

[0047] like Figure 2 As shown in "Decode2", these draft tokens (illustrated as follows) (etc.) are input into subsequent Transformer layers (from arrive The verification computation is performed in parallel on downstream devices using a masked attention mechanism.

[0048] Meanwhile, the main inference path (i.e., the path without using drafts) continues to be calculated, eventually reaching the [number]th [stage]. The layer receives a "real" token. .

[0049] 4. Verification and Rollback Module 240 (Comparison and Decision) This module adjudicates the results of parallel verification: Hit: such as Figure 2 As shown in step 4, if the main path generates a real token Existence A collection of draft tokens China (i.e.) If the draft token is successfully obtained, the speculation is successful. The system accepts the draft token and its corresponding hidden state, and proceeds directly to the next step. Decoding, thus skipping The subsequent layers are computed sequentially.

[0050] Mismatch: such as Figure 2 As shown in step 5, if Not in the draft collection If the speculation fails, the system discards all [the data / information]. The calculation result of the draft token is rolled back to the main path at the specified location. Layer generation and its state, and based on Recalculate .

[0051] Because this mechanism always follows the main path Therefore, this system can guarantee that the output results are completely consistent with the standard (non-speculative) decoding, achieving lossless generation quality.

[0052] 5. Feedback Update Module 250 Based on the results of the verification and rollback module 240, this module updates the strategy of the dynamic decision module 210 in a feedback-driven manner.

[0053] Specifically, it adjusts the entropy threshold To indirectly control the triggering layer : If the speculation is successful, it indicates that the current strategy is too conservative, and the system will improve. (For example, in) walking This makes it easier to trigger speculation at a shallower level (earlier) next time.

[0054] If speculation fails, it indicates that the current strategy is too aggressive, and the system will reduce its risk level. This allows speculation to be triggered later (at a deeper level), reducing the rollback overhead caused by invalid drafts.

[0055] System workflow and optimization modeling The system workflow of this invention is an organic combination of the above modules, and its ultimate optimization goal is to minimize the total end-to-end latency. And satisfy load balancing constraints.

[0056] Optimization Objective The joint optimization objective of the system can be formalized as: ; Constraints (st): 1. (Entropy threshold range) 2. (Trigger layer definition) 3. (Number of drafts) 4. (Load imbalance constraint) 5. (Memory constraints) Delayed modeling Prefill stage delay : During the Prefill phase, the input sequence Divided into Segment, length of each segment . No. The latency of the device is: ; in and These are the calculations of FLOPs for single-layer and embedded layers, respectively. This indicates that the embedding layer is only present on the first device. Total Prefill latency. It is pipeline execution delay and communication delay. The sum of, of which To hide the state dimension.

[0057] Decode stage expected delay : During the Decode phase, there is no speculative delay. Calculate the sum of latency and communication latency for each device.

[0058] ; With the introduction of speculation, the expected delay is: ; in: Is Layers The hit rate of each draft.

[0059] It is the time saved by a successful speculation.

[0060] Parallel verification The additional computational overhead of each draft token.

[0061] It is the rollback cost when speculation fails (e.g.) ).

[0062] Illustration like Figure 1 As shown, the inference time of traditional pipelined parallelism (PP) and the present invention (EdgeSpec) are compared.

[0063] In traditional production line (PP), there are a large number of pipeline "bubbles" (gray areas) in both the Prefill and Decode stages, with equipment idling and waiting.

[0064] In the EdgeSpec of this invention, the Decode phase enables early exits (i.e., hierarchical speculative decoding) at intermediate layers (e.g., determined by low-entropy regions), parallelizing computation and verification in subsequent layers and significantly overlapping computation, communication, and entropy prediction times. This allows the system to save substantial latency (e.g., saving 25% to 75%), reduce idle bubbles, thereby significantly lowering end-to-end latency and improving device utilization.

[0065] like Figure 3 and Figure 4 As shown, the adaptive adjustment capability of the online inference module 200 is demonstrated: like Figure 3 As shown in (a), the triggering layer With decoding steps The system is dynamically changing and will automatically advance or postpone speculation based on the prediction confidence level.

[0066] like Figure 3 As shown in (b), when the equipment utilization rate If the time suddenly increases (e.g., in steps 20-40), the system (through feedback update module 250) will respond dynamically, resulting in a slight increase in decoding time. This is the result of the system actively adjusting its strategy (e.g., postponing speculation) to avoid frequent rollbacks, demonstrating the robustness of the system.

[0067] like Figure 4 As shown, different entropy thresholds This will affect usage frequency, inference time, and hit rate. This system dynamically adjusts these parameters. The aim is to find the optimal balance point among the three (e.g. (At the same time), to achieve the overall optimal combination of utilization, hit rate and latency.

[0068] Application Examples The working process of the system of the present invention will be illustrated below through a specific scenario.

[0069] Suppose an edge collaborative reasoning system consists of three heterogeneous devices (D1: high-performance PC, D2: Jetson Xavier, D3: Jetson Nano) to perform a dialogue question answering task using an LLaMA2-7B model.

[0070] 1. Offline stage: Offline preprocessing module 100 is started.

[0071] The resource awareness and model partitioning module 110 collects computing power, video memory (e.g., D3 has the least video memory), and bandwidth information for D1, D2, and D3.

[0072] Module 110 is based on video memory constraints (especially D3 constraints). Enumerate feasible model partitioning schemes. ,For example .

[0073] The speculative strategy modeling module 120 loads the historical trajectory of LLaMA2-7B and fits it. The Sigmoid model.

[0074] Combined strategy search module 130 , Given load balancing constraints, the optimal initial strategy is obtained through search. ,For example , , .

[0075] 2. Online Phase: When a user initiates a dialogue request in D1, the online inference module 200 is launched.

[0076] The system first presses During the Prefill phase, the key-value cache is initialized.

[0077] Enter the Decode stage to generate the first... tokens: Step S2.1 (Entropy Prediction): Calculate the entropy flowing through D1 The Kalman filter of the dynamic decision module 210 begins prediction. Entropy of the layer.

[0078] Step S2.2 (Threshold Adjustment): Assume the current device load near , Stay nearby.

[0079] Step S2.3 (Trigger Layer Selection): When the calculation reaches... At that time, module 210 predicted Therefore, the triggering layer is determined to be... This layer is located on device D2.

[0080] Step S2.4 (Draft Generation): When the pipeline calculation reaches the 20th layer on D2, the layered speculation module 220 is activated, generating... draft token .

[0081] Step S2.5 (Parallel Verification): These 3 draft tokens are immediately (or in batches) sent to the remaining layers (21-22) of D2 and D3 (23-32) for parallel verification (Decode2). Meanwhile, the main path (Decode1) continues computation on D2 and D3 to obtain... .

[0082] Step S2.6 (Verification): Assuming the main path Exactly equal to The verification and rollback module 240 was determined to have "hit". The system accepted the request. As the final output, the hidden state generated by parallel verification is reused.

[0083] Step S2.7 (Update): Feedback update module 250 receives a "hit" signal, in When walking, Increase (e.g., to 4.1) to encourage earlier layers (such as...) This triggers speculation.

[0084] (Another scenario): Suppose in During the step, the load on device D3 suddenly increased ( (Soaring).

[0085] Module 210 according to Dynamically adjusted upwards (For example, up to 5.0).

[0086] This causes the trigger layer Delayed (e.g. until) (This is because a deeper layer is needed to satisfy the requirement.) conditions.

[0087] This reduces the pressure of parallel verification on D3, avoids frequent rollbacks caused by D3 performance bottlenecks, and ensures the stability of the system under dynamic loads.

[0088] In summary, this invention combines offline preprocessing and online adaptive inference to achieve entropy-aware model partitioning, hierarchical speculative decoding, and dynamic resource scheduling, effectively solving the problems of high latency and load imbalance in large language model inference in heterogeneous edge environments.

[0089] Table 1 summarizes the end-to-end generation latency (in seconds) of EdgeSpec and several representative baseline methods under different models, bandwidths, and deployment environments. The comparison methods include M-LM, Galaxy, EdgeShard, Medusa, and SP. In the EnvA + LLaMA2-7B scenario, EdgeSpec reduces the end-to-end latency from 288.1s (M-LM) to 16.8s at 100Mbps bandwidth, achieving a speedup of 17.1×. In the EnvB + LLaMA2-13B scenario, EdgeSpec achieves a latency of only 43.6s at 500Mbps bandwidth, approximately 81.7% lower than M-LM (237.8s). Compared to the speculative decoding method Medusa, EdgeSpec further reduces latency by approximately 10%–15% in all scenarios. Compared to EdgeShard, EdgeSpec significantly shortens end-to-end latency in heterogeneous device scenarios, with a maximum reduction of 35%. EdgeSpec significantly outperforms existing methods across various device combinations and bandwidth environments, with its performance advantage being particularly pronounced in low-bandwidth or heterogeneous environments.

[0090]

[0091] Table 1 End-to-end generation delay for different algorithms The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A distributed speculative decoding system for edge devices, characterized in that, include: The offline preprocessing module (100) is used for: Based on the resource information of the edge device cluster and the model characteristics of the target large language model, a layer partitioning strategy for the model on the edge device is generated; as well as Based on historical decoding trajectories, initial speculative decoding parameters are determined, wherein the initial speculative decoding parameters include at least an initial entropy trigger threshold; The online inference module (200), connected to the offline preprocessing module, is used for: In the decoding step, inference is performed on the edge device according to the layer partitioning strategy; During the inference process, the prediction distribution entropy of the intermediate layer of the model is dynamically predicted; Based on the predicted entropy and a dynamically adjusted entropy trigger threshold, a speculative triggering layer for launching drafts is determined. When the reasoning reaches the speculation trigger layer, at least one draft token is generated; as well as On the downstream edge device following the speculative triggering layer, the at least one draft token is verified in parallel to obtain the final decoded token.

2. The system according to claim 1, characterized in that, The online inference module (200) is also used for: Real-time monitoring of the device utilization rate of the edge devices; and The entropy trigger threshold is dynamically adjusted based on the deviation between the real-time device utilization rate and a reference utilization rate.

3. The system according to claim 2, characterized in that, The online inference module (200) dynamically adjusts the entropy trigger threshold in the following manner: When the real-time utilization rate is higher than the reference utilization rate, the entropy trigger threshold is increased to trigger speculation at a layer with lower entropy. When the real-time utilization rate is lower than the reference utilization rate, the entropy trigger threshold is reduced to trigger speculation at a layer with higher entropy.

4. The system according to claim 1, characterized in that, The online inference module (200) predicts the predicted distribution entropy of the intermediate layer using a Kalman filter.

5. The system according to claim 1, characterized in that, The offline preprocessing module (100) is also used for: The layer partitioning strategy, the initial entropy trigger threshold, and the number of candidate drafts are jointly optimized, with the optimization objective being to minimize the estimated end-to-end latency and device load imbalance.

6. The system according to claim 1, characterized in that, The online inference module (200) is also used in the parallel verification for: Obtain the final decoded token calculated from the main path; Compare whether the final decoded token is in the set of at least one draft token; If so, it is determined to be a speculative hit, and the parallel computation result corresponding to the draft token is accepted; If not, it is determined to be a speculative mismatch, and the result is rolled back to the main path calculation result.

7. The system according to claim 6, characterized in that, The online inference module (200) is also used for: When the speculative mismatch occurs, the entropy trigger threshold is adjusted in subsequent decoding steps to postpone speculative triggering. When the speculative hit occurs, the entropy trigger threshold is adjusted in subsequent decoding steps to trigger the speculative hit earlier.

8. The system according to claim 1, characterized in that, The offline preprocessing module (100), when determining the initial speculative decoding parameters, is also used for: By fitting a Sigmoid model, the relationship between draft hit rate and entropy and layer depth under different layers and different numbers of candidate drafts is modeled.

Images