An end-cloud collaborative speculative decoding acceleration method based on credit inertia

Abstract

The application discloses an end-cloud collaborative speculative decoding acceleration method based on credit inertia, and belongs to the field of edge-cloud collaborative computing. The method improves the efficiency and stability of end-cloud collaborative generation through credit inertia perception, adaptive threshold prediction and asynchronous parallel scheduling. Firstly, the Jensen-Shannon divergence is used to measure the distribution difference between the draft model and the cloud target model, and a robust distribution-level verification criterion is constructed. Then, the "credit inertia" feature of the historical window is extracted through the exponential weighted moving average, and the acceptance threshold and the draft window length are adaptively adjusted through a lightweight threshold prediction network. Finally, the gradual preview and concurrent mechanism is introduced, and the draft generation, cloud pre-trial and full trial are decoupled into an asynchronous pipeline, thereby significantly reducing the end-to-end delay, reducing invalid rollback, and improving the effective token throughput rate under the premise of ensuring the generation quality, and being suitable for high real-time applications such as mobile album, multi-modal retrieval and description generation.

Description

Technical Field

[0001] This invention belongs to the field of edge-cloud collaborative computing, and more specifically, relates to an edge-cloud collaborative speculative decoding acceleration method based on credit inertia. Background Technology

[0002] With the widespread deployment of large-scale models in critical business applications such as intelligent customer service, code generation, and traffic scheduling, interactive applications for end users are rapidly shifting from "cloud-centralized" to "edge-cloud collaborative." In this process, the latency, energy consumption, and cost of large language model inference directly impact business experience and system scalability. Due to the massive parameter scale and computationally intensive nature of large cloud-based models, relying solely on cloud-side inference often fails to meet the real-time response requirements of massive concurrent requests; while deploying the complete model to the edge or mobile devices is subject to multiple constraints related to computing power, storage, and power consumption. Therefore, how to achieve low-latency and high-precision generative services through collaborative inference between lightweight small models and large cloud models within an edge-cloud collaborative framework has become a key issue in current intelligent application infrastructure.

[0003] Among numerous acceleration technologies, speculative decoding introduces a "draft-verification" dual-model structure, providing a new acceleration path for large-model inference at the system level: a small model on the edge or near-end generates a sequence of candidate tokens, which is then verified and corrected by the large model in the cloud, thereby improving decoding throughput without significantly sacrificing accuracy. However, most existing speculative decoding schemes are designed for single-machine or homogeneous cluster scenarios, implicitly assuming that computing and communication resources are relatively abundant and stable. When this paradigm is migrated to a real-world edge-cloud collaborative environment, factors such as limited uplink bandwidth, significant network latency fluctuations, and highly heterogeneous edge computing power amplify the risks caused by the distribution deviation between the draft model and the target model: once the draft sequence is rolled back on a large scale, not only is the expected acceleration benefit lost, but additional recalculation and communication overhead are also introduced, leading to an increase in end-to-end latency.

[0004] On the other hand, existing methods typically rely on pre-set fixed acceptance thresholds or simplified statistical assumptions to determine whether a draft token can be "accepted" by the target model. These static thresholds exhibit significant rigidity issues under real-world business traffic: when the input context, task type, or network conditions change, overly conservative thresholds frequently reject high-quality drafts that would otherwise be acceptable, reducing effective throughput; while overly aggressive thresholds lead to error accumulation and large-scale rollbacks, affecting generation quality. In scenarios such as continuous dialogue, multi-turn interactions, and long text generation, the patterns of draft acceptance or rejection often have a clear temporal correlation, but this cross-window statistical information and "inertia" characteristics have not yet been systematically mined and utilized, making it difficult for edge-cloud collaborative inference decoding to adaptively adjust strategies based on historical performance.

[0005] Meanwhile, from a system perspective, existing edge-cloud collaborative inference frameworks often exhibit a clear "mutual waiting" phenomenon between draft generation and cloud verification: the edge needs to wait for the verification results of the previous batch of tokens to safely advance the context, while the cloud relies on the edge to upload a complete draft sequence to initiate efficient parallel verification; coupled with unstable network transmission latency, this easily creates a performance bottleneck in the inference pipeline. The lack of joint modeling and dynamic scheduling of edge-side computation, cloud-side load, and network conditions also makes it difficult for the system to simultaneously achieve multi-dimensional goals such as response time, cloud energy consumption, and service quality in different business scenarios. Summary of the Invention

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides an edge-cloud collaborative speculative decoding acceleration method based on credit inertia, thereby solving the performance bottlenecks caused by fixed threshold rigidity, mutual waiting between edge and cloud, and network latency fluctuations in traditional edge-cloud speculative decoding.

[0007] To achieve the above objectives, according to a first aspect of the present invention, an edge-cloud collaborative speculative decoding acceleration method based on credit inertia is provided, comprising:

[0008] S1, send the draft window consisting of the 1st to Wth tokens predicted by the draft model on the edge side as the current decoding window to the cloud side; at the same time, the draft model continues to make predictions; W>1;

[0009] The input to the draft model is a sequence of numbers corresponding to the natural language text input by the user;

[0010] S2, the target model on the cloud side verifies the first token in the current decoding window according to the verification threshold of the current decoding window. If the verification is successful, the target model verifies the other tokens in the current decoding window in sequence according to the verification threshold of the current decoding window. Otherwise, the target model generates a full-window rejection signal, discards all tokens in the current decoding window, resamples and generates a new token, and sends it along with the full-window rejection signal to the draft model. The draft model updates the context according to the new token, and the verification of the current decoding window ends.

[0011] In this process, when the target model sequentially verifies other tokens in the current decoding window according to the verification threshold, if a token fails verification, it discards that token and the remaining tokens in the current decoding window, resamples and generates a new token, and sends it along with the sequence number of the verified tokens to the draft model. The draft model discards all tokens after the verified token and appends the resampled token to the end of the verified tokens, updates the context, and the verification of the current decoding window ends. If all tokens in the current decoding window pass verification, the target model sends the entire window's receive signal to the draft model, and the verification of the current decoding window ends.

[0012] S3, sequentially take the draft window composed of the W+1~2Wth tokens predicted by the draft model, the draft window composed of the 2W+1~3Wth tokens, ..., the draft window composed of the AW+1~(A+1)×Wth tokens as the current decoding window and return to S2, until the termination condition is met; A>1.

[0013] According to a second aspect of the present invention, an electronic device is provided, comprising: a computer-readable storage medium and a processor;

[0014] The computer-readable storage medium is used to store executable instructions;

[0015] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect.

[0016] According to a third aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to perform the method as described in the first aspect.

[0017] According to a fourth aspect of the invention, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the method described in the first aspect.

[0018] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0019] This invention provides a credit-inertia-based edge-cloud collaborative speculative decoding acceleration method for accelerating edge-cloud collaborative inference in natural language generation tasks using text data. The method includes: collecting user-inputted text to be processed, encoding it into a token sequence, and inputting it into an edge-cloud collaborative inference system composed of a pre-trained lightweight draft model on the edge and a target large model in the cloud; utilizing a credit-inertia-driven adaptive threshold control and a progressive parallel decoding mechanism to perform segmented speculation and verification of the generation process, ultimately outputting a coherent and reliable generated text result; this method has the following advantages:

[0020] 1. The method provided by this invention, in order to minimize latency, decouples the synchronous dependency between draft generation and cloud verification. The cloud verification process is decomposed into two asynchronous stages: pre-review and full review. In the pre-review stage, the target model in the cloud only verifies the first token in the current decoding window. If the verification fails, the entire window is immediately rejected and a regeneration signal is sent to the draft model to avoid invalid transmission. If the verification passes, the process enters the full review stage, where the target model in the cloud asynchronously verifies the remaining tokens in the current decoding window, while the draft model continues to generate the next draft window. This achieves time overlap execution on both the edge and cloud sides, which can reduce end-to-end latency and improve resource utilization.

[0021] 2. As a further preferred embodiment, the method provided by this invention transforms discrete token verification into a continuous distribution measurement task, and uses Jensen-Shannon divergence to characterize the symmetrical difference between the output distribution of the draft model and the output distribution of the target model, which can further improve the decoding quality.

[0022] 3. As a further preferred embodiment, the method provided by this invention, considering that speculative decoding based on a fixed verification threshold is prone to excessive rejection of high-quality draft tokens, adopts an adaptive verification threshold switching method based on credit inertia. The smoothed pass rate of the K historical decoding windows adjacent to the current decoding window is used as a credit score. The verification threshold of the current decoding window is predicted based on this credit score and the instantaneous pass rate of the K historical decoding windows adjacent to the current decoding window. Simultaneously, a hysteresis control strategy is employed: the verification threshold is only set if the difference between the predicted value and the current value exceeds a set value. Furthermore, an update is only triggered after P historical decoding windows have been completed, thus avoiding performance fluctuations caused by frequent oscillations. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the edge-cloud multimodal adaptive inference decoding method based on credit inertia provided in an embodiment of the present invention.

[0024] Figure 2 The training flowchart of the threshold predictor provided in the embodiment of the present invention is shown. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0026] In edge-cloud collaborative large-scale model inference scenarios, mutual waiting between edge and cloud devices and network latency fluctuations cause high inference latency and insufficient throughput. To address this, this invention provides an edge-cloud collaborative speculative decoding acceleration method based on credit inertia, used for edge-cloud collaborative accelerated inference of text data in natural language generation tasks. User input can be a dialogue query, code completion request, or document summary instruction. The method includes:

[0027] S1, the draft window consisting of the first to W tokens predicted by the draft model on the edge side (i.e., the first draft window) is sent to the cloud side as the current decoding window; at the same time, the draft model continues to make predictions; W>1;

[0028] The input to the draft model is a sequence of numbers corresponding to the natural language text input by the user;

[0029] S2, the target model on the cloud side verifies the first token in the current decoding window according to the verification threshold of the current decoding window. If the verification is successful, the target model verifies the other tokens in the current decoding window in sequence according to the verification threshold of the current decoding window. Otherwise, the target model generates a full-window rejection signal, discards all tokens in the current decoding window, resamples and generates a new token, and sends it along with the full-window rejection signal to the draft model. The draft model updates the context according to the new token, and the verification of the current decoding window ends.

[0030] In this process, when the target model sequentially verifies other tokens in the current decoding window according to the verification threshold, if a token fails verification, it discards that token and the remaining tokens in the current decoding window, resamples and generates a new token, and sends it along with the sequence number of the verified tokens to the draft model. The draft model discards all tokens after the verified token and appends the resampled token to the end of the verified tokens, updates the context, and the verification of the current decoding window ends. If all tokens in the current decoding window pass verification, the target model sends the entire window's receive signal to the draft model, and the verification of the current decoding window ends.

[0031] S3, sequentially taking the draft window composed of the (W+1)th to (2W)th tokens predicted by the draft model (i.e., the 2nd draft window), the draft window composed of the (2W+1)th to (3W)th tokens (i.e., the 3rd draft window), ..., the draft window composed of the (AW+1)th to (A+1)×Wth tokens (i.e., the Ath draft window) as the current decoding window and returning to S2, until the termination condition is met; A>1.

[0032] In step S1, the user-input natural language text is first acquired, encoded by a token segmenter (the length of the encoded token sequence is variable), and then mapped to a corresponding list of numbers, i.e., a sequence of digits. This sequence of digits serves as a lightweight draft model on the context input edge side for autoregressive prediction, generating a token sequence. The draft model also outputs the probability distribution of each token, which is used for subsequent distribution measurement and credit assessment.

[0033] The edge-cloud collaborative system supports both batch processing and streaming generation modes.

[0034] The method provided by this invention decouples the draft generation and cloud verification from synchronous dependency to minimize latency. In traditional methods, the draft model waits for verification feedback, and the target model waits for the complete draft sequence, resulting in an overlap of uplink transmission latency and computation waiting time. Therefore, in step S2, the method provided by this invention employs a Progressive Lookahead Concurrency (PLC) mechanism, decomposing the cloud verification process into two asynchronous stages: pre-review and full review. In the pre-review stage, the target model in the cloud only verifies the first token in the current decoding window t. If the verification fails, the entire window will be rejected immediately and a regeneration signal will be sent. The token is sent to the draft model, and the target model resamples a new token and sends it to the draft model for a full window rollback to avoid invalid transmissions. If the verification passes, the process enters the full review stage, where the target model in the cloud asynchronously verifies the remaining tokens in the current decoding window. Meanwhile, the draft model continues to generate the next draft window.

[0035] The above process maximizes throughput through time overlap:

[0036]

[0037] in, and Hiding the overhead through pipeline parallelism, only retaining the communication overhead. Regeneration time after rejection .

[0038] To ensure KV cache consistency, the draft model maintains a dual buffer: the main buffer stores verified prefixes, and the shadow buffer stores speculative continuations. When the draft model receives a rejection signal, it discards the shadow buffer and restores its state from the main buffer; when it receives an acceptance signal, it merges the two buffers and updates the main buffer pointer. By constructing such a complete asynchronous collaborative mechanism, this method integrates the decentralized, serial verification process into a compact, parallel inference pipeline, thereby building an efficient, low-latency edge-cloud collaborative closed loop.

[0039] Specifically, such as Figure 1 As shown, to further reduce end-to-end latency and improve resource utilization, this invention is based on a Progressive Lookahead Concurrency (PLC) mechanism, which decouples draft generation and cloud verification into two stages: pre-review and full review, achieving overlapping execution on both the edge and cloud sides. The two stages are further explained below.

[0040] (1) Pre-screening stage: rapid screening of the first token

[0041] In the PLC mechanism, when the draft model on the edge generates a new draft window of length K, it immediately uses this window as the current decoding window t and sends it along with its corresponding distribution information to the cloud. The target model on the cloud prioritizes fast verification of the first token in the current decoding window t. The cloud calculates the distribution divergence of this first token and compares it with the verification threshold of the current decoding window t. Comparison:

[0042] If the distribution divergence of the first token is greater than If the window is deemed to be of poor overall quality, the cloud does not need to continue verifying the remaining tokens and immediately generates a rejection signal to return to the edge side.

[0043] If the distribution divergence of the first token is less than or equal to If the window is deemed to have a high success rate, it will proceed to the full review stage to fully verify the remaining tokens in the current decoding window t.

[0044] This "first token pre-screening" strategy can eliminate a large number of poor-quality draft windows generated by the edge-side draft models in a very short time, avoiding invalid calculations and transmissions of the remaining tokens.

[0045] Understandably, when validating the first draft window generated by the draft model, t=1; when validating the second draft window generated by the draft model, t=2, and so on.

[0046] (2) Full review stage: asynchronous verification and draft preview generation

[0047] Once the current decoding window t passes the pre-approval, the cloud initiates a full review process to measure the distribution and determine acceptance of the remaining tokens in that window. During this period, the edge-side draft model no longer waits for the verification result of the current decoding window, but continues to generate the next draft window based on the provisional context, forming a prospective draft sequence.

[0048] During this process, the system maintains two sets of key-value caches: the main cache and the shadow cache.

[0049] The main buffer stores the token prefix that has been finally confirmed by the cloud (i.e., the draft window that has been verified by the cloud); the shadow buffer stores speculative continuations based on windows that have not yet been confirmed. When the cloud completes the full review of the current window, if the window is accepted, an acceptance signal is returned, and the edge side merges the content of the shadow buffer into the main buffer and updates the context; if the window is rejected, the corresponding token in the shadow buffer is discarded according to the rejection signal, a token resampled by the target model is received and merged into the end of the verified token sequence, and the draft model continues to generate subsequent tokens based on this sequence. Through this double-buffering mechanism, the system maximizes the parallelism between draft generation and cloud verification while ensuring context consistency.

[0050] Understandably, when validating the first draft window generated by the draft model, t=1; when validating the second draft window generated by the draft model, t=2, and so on.

[0051] For example, assuming the length W=5 of each draft window (i.e., each draft window consists of 5 tokens), when validating the second draft window generated by the draft model (consisting of the 6th to 10th tokens generated by the draft model):

[0052] If the first token in the second draft window (i.e., the 6th token generated by the draft model) fails verification, the target model generates a full-window rejection signal and sends it to the cloud. At the same time, it discards all tokens in the second draft window and resamples and generates a new token based on the already verified context (i.e., the 1st to 5th tokens) and sends it to the edge. The draft model at the edge updates the context in the shadow buffer: the draft model discards all tokens in the shadow buffer of the second draft window (i.e., the 6th to 10th tokens originally generated by the draft model) and all tokens in all draft windows after the second draft window, and uses the new token as the new 6th token to regenerate subsequent tokens.

[0053] Assuming that the first and second tokens (i.e., the 6th and 7th tokens generated by the draft model) in the second draft window are both verified, but the third token (i.e., the 8th token generated by the draft model) in the second draft window fails verification, the target model generates a partial rollback signal and sends it to the edge. Based on the verified context (i.e., the 1st to 7th tokens), it resamples and generates a new token, which is then sent to the draft model along with the sequence numbers (i.e., 1st and 2nd) of the verified tokens in the second draft window. The draft model discards the 3rd to 5th tokens in the second draft window based on this signal and updates the context: the draft model discards all tokens after the verified tokens (i.e., the 1st to 7th tokens), takes the new token sent from the cloud as the 8th token, and continues to generate subsequent draft tokens based on the existing ones.

[0054] While cloud-based verification can guarantee generation quality, traditional methods mostly rely on negative sampling using the confidence ratio of draft and target tokens. However, traditional negative sampling strategies based on confidence ratios only focus on the probability of a single token, ignoring the global consistency of the draft-target distribution. This leads to the incorrect acceptance of locally high-quality but globally skewed tokens. To address this issue, the method provided in this invention transforms discrete token verification into a continuous distribution measurement task. It employs the Jensen-Shannon divergence to characterize the symmetrical difference between the output distribution of the draft model and the output distribution of the target model, establishing a more robust verification criterion. This process utilizes a distribution measurement function... To achieve this, that is, as a further preferred solution, in step S2, if the current decoding window's... The distribution divergence of each token If the value is less than or equal to the verification threshold of the current decoding window, the verification passes; otherwise, the verification fails.

[0055] Among them, the first in the current decoding window The distribution divergence of each token The calculation formula is:

[0056] ;

[0057] in, and Let be the probability distributions of the i-th token output by the draft model and the target model, respectively. It is a mixed distribution. , In mixed distribution Down of divergence, In mixed distribution Down of Divergence; for and The distribution similarity calculation function, for and JS divergence.

[0058] This approach effectively upgrades the estimation method from point estimation to distribution estimation.

[0059] The method provided by the present invention can set the verification threshold of each current decoding window to the same fixed threshold, and verify the token of each current decoding window according to the fixed threshold in step S2.

[0060] Traditional methods often use fixed thresholds to verify the similarity between draft and target token distributions, which frequently leads to the excessive rejection of high-quality tokens due to neglecting temporal dependencies, making them difficult to apply in mobile scenarios with stringent real-time requirements. Research has found that when statistically analyzing the window pass rate in the edge-cloud inference decoding process, on datasets such as MMLU and QuALITY, the token pass rate of consecutive windows does not fluctuate randomly, but rather exhibits a pattern of maintaining a "high pass rate range" or "low pass rate range" for extended periods, with state transitions occurring only in a few locations. This indicates that the performance of the current window is highly correlated with the performance of the preceding window, exhibiting a clear "credit inertia": once a high-credit state is reached, subsequent windows tend to continuously improve from "good" to "good"; conversely, when in a low-credit state, they tend to continuously improve from "bad" to "bad." Based on this, as a further preferred solution, the method provided in this invention compresses the pass rates of the K adjacent historical decoding windows (i.e., the K verified decoding windows adjacent to the current decoding window) into credit scores, and adaptively adjusts the verification threshold of the current decoding window t according to these scores. The verification threshold is appropriately relaxed during the high-credit phase to improve throughput, and tightened during the low-credit phase to suppress invalid drafts and rollbacks, thus forming a cloud-edge speculative decoding strategy where the verification threshold of the decoding window is dynamically adjusted over time. Specifically, preferably, in step S2, before the target model on the cloud side verifies the first token in the decoding window based on the current verification threshold, the following steps are also included:

[0061] (1) Calculate the instantaneous pass rate of the K historical draft windows (i.e. the K verified decoding windows adjacent to the current decoding window t) t-1, t-2, ..., tK. ;

[0062] in, , , Let be the pass rates of the K historical decoding windows t-1, t-2, and tK adjacent to the current decoding window t, respectively; for ease of description, use . To provide a unified expression For historical decoding window The pass rate, j = t-1, t-2, ..., tK, , For historical decoding window The Middle The distribution divergence of each token For historical decoding window The verification threshold; For indicator functions, hour, ,otherwise ;

[0063] (2) Calculate the average pass rate of the K historical decoding windows t-1, t-2, ..., tK adjacent to the current decoding window t. ;

[0064] in, = ;

[0065] (3) According to Calculate the smoothing pass rate of the K historical decoding windows t-1, t-2, ..., tK adjacent to the current decoding window t. ;

[0066] in, , These are the weighting coefficients. The smoothed pass rate of the K historical decoding windows t-2, t-3, ..., t-1-K adjacent to the previous decoding window t-1;

[0067] (4) and The input is fed into a pre-trained predictor to obtain the verification threshold for the current decoding window t. Predicted value ,like and If the absolute value of the difference exceeds the set value, and the absolute value of the difference between the predicted value and the verification threshold of each of the P previous historical decoding windows adjacent to the current decoding window exceeds the set value, then... The value is updated to The value, otherwise The value remains unchanged.

[0068] Specifically, in edge-cloud speculative decoding practices, the token pass rates of adjacent decoding windows often exhibit strong temporal correlation. That is, draft windows with recent high pass rates (e.g., greater than 0.8) are likely to maintain high performance, while draft windows with recent low pass rates (e.g., less than 0.4) continue to perform poorly. This "credit inertia" is a key clue for adaptive verification, but traditional methods lack systematic modeling of historical patterns. Considering that fixed-threshold speculative decoding easily leads to excessive rejection of high-quality draft tokens, this invention provides a method based on credit inertia: when the token pass rate of the most recent historical window draft is high, the verification threshold is appropriately relaxed; when the token pass rate of the most recent historical window draft is low, the verification threshold is made more stringent, thereby reducing the rejection rate. Exponentially weighted moving average (EWMA) is used to capture temporal dynamics and generate a smooth pass rate estimate.

[0069] First, calculate the average pass rate of the K historical decoding windows adjacent to the current decoding window (i.e., the K verified decoding windows adjacent to the current decoding window). :

[0070]

[0071] in, Let W be the distribution divergence of the i-th token in the j-th historical decoding window, and let W be the size of the draft window (i.e., the size of the current decoding window and the size of the historical decoding window, both of which are W). For indicator functions, hour, ,otherwise .

[0072] Then, to avoid the drastic impact of fluctuations in a single window on the strategy, an exponentially weighted moving average mechanism is used to smooth the average pass rate of K historical decoding windows into a credit inertia estimate. That is, based on EWMA is used to calculate the smooth pass rate of the K historical decoding windows adjacent to the current decoding window t. :

[0073]

[0074] in, is a weighting coefficient, ranging from (0,1), used to balance short-term responsiveness and long-term stability. The smooth pass rate of the current decoding window t. The instantaneous pass rate of the K historical decoding windows t-1, t-2, ..., tK adjacent to the current decoding window. The compressed representation of the credit score, accordingly, The pass rate for the historical decoding window t-1. The pass rate for the historical decoding window t-2. For historical decoding window The pass rate.

[0075] By employing the above method, a balance is struck between responsiveness to short-term fluctuations and stability of long-term trends, transforming discrete verification events into continuous state signals and providing high-quality input features for threshold prediction.

[0076] Finally, the smoothed pass rate of the K historical decoding windows adjacent to the current decoding window t is calculated. With real-time pass rate The common inputs are fed into a pre-trained predictor to obtain the verification threshold for the current decoding window t. Predicted value ,like and If the absolute value of the difference exceeds the set value, and the predicted values ​​of the verification thresholds of the P previous historical decoding windows adjacent to the current decoding window are all greater than the verification thresholds of the P previous verified decoding windows, then... The value is updated to The value, otherwise The value remains unchanged.

[0077] The method provided by this invention employs an online stability guarantee mechanism and introduces a hysteresis control strategy: only when the difference between the predicted value and the current value of the verification threshold exceeds a set value... Furthermore, updates are only triggered after P historical decoding windows have been completed, avoiding performance fluctuations caused by frequent oscillations. This is achieved by adopting... With respect to the parameters P, the method provided by this invention can significantly reduce the disturbances caused by frequent adjustments to the peer cloud pipeline while ensuring adaptability, thereby improving the stability and predictability of the overall inference process.

[0078] For example, assuming P=3, then for the current decoding window t, when its predicted value of the verification threshold is... If the absolute value of the difference between the current value and the previous value exceeds a set value, the verification threshold value of the current decoding window t will only be updated if the absolute value of the difference between the predicted value and the current value of the verification threshold of each of the historical decoding windows t-1, t-2, and t-3 exceeds the set value. The value of t is not specified; otherwise, the verification threshold of the current decoding window t remains unchanged.

[0079] It is understandable that when decoding the first to Kth draft windows of the draft model, the validation threshold is the initial value. It is not until the (K+1)th draft window is decoded that the determination of whether to adaptively update the validation threshold is involved.

[0080] To address the computational overhead of real-time calculation of the optimal threshold in online inference, the method provided in this invention employs a data-driven mapping learning strategy. For example... Figure 2 As shown, a nonlinear mapping from the instantaneous pass rate and smooth pass rate of the K historical decoding windows adjacent to the decoding window to the optimal verification threshold of the decoding window is constructed through offline calibration, and a neural network-based predictor (also known as a threshold predictor) is trained to achieve fast prediction of the verification threshold.

[0081] The training process of the predictor includes:

[0082] (1) Constructing the training dataset: Set the validation threshold of all decoding windows to a given threshold T, decode the natural language text samples input by the user, and calculate the validation threshold of each decoding window under different given threshold T configurations. The joint cost is used to select the N decoding windows with the smallest joint cost. For each of the N decoding windows, the smooth pass rate and instant pass rate of the K historical decoding windows adjacent to it, along with the corresponding given threshold, are used as the training dataset.

[0083] Among them, each decoding window Joint cost under a given threshold T configuration ; express End-to-end latency, including draft model generation, under a given threshold T configuration. Time and target model validation The time required for the verification results to be returned to the draft model; express The pass rate under a given threshold T configuration; This is a weighting factor.

[0084] (2) The predictor is trained with the smooth pass rate and instant pass rate of the K historical decoding windows adjacent to the decoding window in the training dataset as inputs, and with the goal of minimizing the difference between the predicted value and the true value (i.e. the given threshold corresponding to the decoding window in the training set) of the decoding window output by the predictor; wherein the predictor is a neural network, for example, a five-layer feedforward network structure can be adopted.

[0085] The predictor is trained using an "offline calibration + online inference" approach, by constructing a training dataset. Complete the learning of the nonlinear mapping so that it can be based on the smooth pass rate of the K historical decoding windows adjacent to the current decoding window t. With real-time pass rate The optimal value of the prediction validation threshold is determined. Specifically, the training process of the threshold predictor is as follows:

[0086] 1) Run log collection

[0087] First, a complete edge-cloud speculative decoding framework was deployed in a representative application scenario and repeatedly run under different network bandwidths, latency, and device types to collect a large number of real requests. The edge-cloud time-overlapping decoding method based on a progressive pre-aiming concurrency mechanism, provided by this invention, was used to decode user-input natural language text samples, processing each decoding window for each request. Record the following information: the given value T of the dynamic threshold, and the smoothed pass rate of the K historical windows adjacent to the decoding window. With real-time pass rate Then calculate each decoding window The combined cost. Continuously change the value of T and repeat the above process.

[0088] In other words, when constructing the training dataset, a balance between end-to-end latency and token pass rate is considered, and a joint cost function is constructed. Used to calculate across all decoding windows When decoding user-input natural language text samples using the edge-cloud time-overlapping decoding method based on progressive pre-aiming concurrency mechanism provided by this invention, with all verification thresholds set to a given threshold T, each decoding window... The joint cost, while recording each decoding window real-time pass rate and smooth pass rate The given threshold T is continuously changed, and the joint cost of each decoding window is recalculated. Using this method, the joint cost of each decoding window under different values ​​of the given threshold T is tested offline in a limited number of natural language question-answering samples. Then, the N decoding windows with the lowest joint cost are selected. For each of these N decoding windows, the smooth pass rate and instant pass rate of its K neighboring historical decoding windows, along with the corresponding given threshold, are used as the training dataset. .

[0089] For example, suppose T uses 5 different values, from which 20 decoding windows with the minimum joint cost are selected. Then, for each of these 20 decoding windows, the smooth pass rate and instant pass rate of its K neighboring historical decoding windows, along with the corresponding given thresholds, are used as the training dataset. .

[0090] The optimal control strategies are summarized under different credit inertia levels, different network conditions, and different task difficulties.

[0091] 2) Predictor Structure Design and Training

[0092] Threshold predictor ( (For predictor parameters) A multi-layer feedforward neural network structure is preferred.

[0093] When training the predictor, first select a decoding window, input the smooth pass rate and instant pass rate of the K adjacent historical decoding windows into the predictor, and record the predicted value of the validation threshold output by the predictor. Calculate the difference between this predicted value and the given threshold corresponding to the decoding window in the training set. Repeat this process until every decoding window in the training set is traversed. Update the predictor parameters through backpropagation to complete the training of the predictor.

[0094] After training is complete, the trained predictor will be... It is permanently deployed in the edge-cloud control module.

[0095] During the online inference phase, the predictor The input is the smoothed pass rate of the K historical decoding windows adjacent to the current decoding window t. and real-time pass rate The output is the verification threshold for the current decoding window t. Predicted value , that is:

[0096]

[0097] The process of generating offline training data is as follows: exhaustively search on representative datasets. The parameters are combined, and their joint cost is evaluated on the validation set for each configuration. The configuration that minimizes the objective function is selected as the supervision label, and the training dataset is constructed. ;

[0098] To maximize inference efficiency, the entire verification process is sensitive to system performance bottlenecks. Finally, all intermediate statistics generated in the first three stages of the credit-inertia-based edge-cloud multimodal adaptive inference decoding framework—from draft to target distribution measurement to the final threshold decision—are seamlessly integrated into a standardized, real-time-scheduling verification strategy. This is implemented through a lightweight predictor that takes all historical verification statistics as input and instructs the neural network to generate a configuration according to a predefined joint optimization objective. The strategy structure is rigorous, including key modules such as request overview, distribution divergence sequence, verification starting point (credit inertia state awareness), adaptive adjustment path (dynamic updating and evaluation of the threshold, highlighting historical evidence of the optimal window size), final pass rate and latency profile, and asynchronous scheduling suggestions and optimization points. By generating such a complete configuration, this method materializes the complex, distributed verification process into a dynamic, traceable policy record, thereby constructing an efficient, adaptive inference closed loop.

[0099] In summary, this invention provides an edge-cloud collaborative inference decoding acceleration method based on credit inertia for edge-cloud collaborative generation of text or multimodal data. The method includes: encoding user input and feeding it into a lightweight draft model on the edge to generate a token sequence; constructing the current decoding window and sending it to the cloud; in the target model on the cloud, calculating the divergence of the token output distribution of the current decoding window based on a distribution metric function, and calculating the credit inertia estimate (i.e., smoothed pass rate) by combining the instantaneous pass rates of the K historical decoding windows adjacent to the current decoding window; using this as input, outputting the predicted value of the verification threshold for the current decoding window through a threshold predictor and determining whether to update the verification threshold; determining whether the token is accepted based on the verification threshold, generating corresponding acceptance or rollback signals, and scheduling draft generation and cloud verification through a progressive edge-cloud asynchronous pipeline mechanism; after decoding meets the termination condition, decoding the token sequence finally confirmed by the cloud into natural language text or a descriptive result as the final output.

[0100] This method employs strategies such as asynchronous verification, distributional metric verification based on JS divergence, and threshold prediction based on historical statistics to systematically guide the lightweight draft model and the cloud-based target model to collaboratively generate high-quality descriptions. By mining the correlation of the draft window's pass rate over time, this method treats it as a "credit signal" in the edge-cloud collaborative inference process. Based on this, it adaptively adjusts the decoding threshold, draft length, and edge-cloud pipeline concurrency strategy to significantly reduce the computational and communication overhead caused by invalid verification and rollback while ensuring generation quality. This provides a scalable technical foundation for the efficient deployment of large models in heterogeneous edge-cloud environments. A dynamic credit score is constructed between the lightweight draft model on the edge and the verification model on the cloud using historical window pass rates, adaptively adjusting the verification threshold to achieve joint constraints on draft quality and edge-cloud communication overhead. At the system level, a progressive edge-cloud parallel pipeline is designed to decouple the draft generation, segmented verification, and feedback submission processes. Asynchronous queues and hardware-aware scheduling reduce invalid rollbacks and blocking waits. Extensive experiments have shown that the method provided by this invention significantly reduces end-to-end latency and improves effective throughput while ensuring generation quality, providing an efficient and scalable solution for accelerating speculative decoding of large models in an edge-cloud collaborative environment.

[0101] The proposed Elastic Edge–Cloud Speculative Decoding (E) method based on credit inertia was demonstrated on multiple benchmark datasets and model combinations. 2 A systematic evaluation was conducted using the SCI (Science and Technology Innovation Center). Specifically, for inference tasks, the GSM8K mathematical reasoning dataset, the CommonsenseQA commonsense question answering dataset, the MMLU large-scale multi-task general knowledge comprehension dataset, and the HumanEval code generation dataset were selected to cover different types of workload scenarios, including mathematical reasoning, fact-based question answering, and comprehensive knowledge and program generation. Regarding models, various mainstream large language models, such as the Qwen2 series, Llama 3.1 series, Gemma2 series, and DeepSeek-R1-Distill-Qwen 1.5B+32B, were used to simulate different scales of "small edge model + large cloud model" combinations to verify the universality and stability of the method under different model architectures and parameter scales.

[0102] The experiment was conducted on a distributed edge-cloud test platform consisting of NVIDIA RTX 4090 and A6000 GPUs. It simulated a typical edge device and cloud server collaborative deployment scenario in a network environment with controllable bandwidth and latency. During inference, the batch size was set to 1 to approximate the per-token generation mode under real online interactive services. Evaluation metrics included: the number of tokens generated and ultimately accepted by the cloud per unit time (throughput, tokens / s), the average inference latency of a single token and the entire output, the speedup ratio under edge-cloud collaboration (relative to the standard speculative decoding baseline), and task accuracy metrics on various datasets (e.g., accuracy of GSM8K, accuracy of CommonsenseQA and MMLU, and pass rate of HumanEval). Comparison results between conventional single-machine scenarios and edge-cloud collaborative scenarios show that the E provided by this invention... 2 While maintaining or even slightly improving task accuracy, SCI achieves approximately 1.8 to 2.4 times the throughput improvement and significant end-to-end latency reduction compared to traditional speculative decoding and fixed threshold methods on different datasets, verifying the effectiveness of credit inertia-based adaptive threshold control and progressive pre-aiming concurrency mechanism in real-world edge-cloud collaborative inference environments.

[0103] Under various typical reasoning tasks and model combinations, the E method provided by this invention, compared with traditional speculative decoding methods that do not use credit inertia mechanisms, offers superior performance. 2 -SCI effectively reduces performance degradation caused by network fluctuations and rigid fixed thresholds in edge-cloud collaborative scenarios. Experimental results show that, under the same task accuracy constraints, the E provided by this invention... 2 -SCI's end-to-end latency is significantly lower than the baseline method, the number of tokens ultimately accepted by the cloud per unit time (effective throughput) is steadily improved, and it exhibits good robustness under different network bandwidth and latency settings.

[0104] Further comparative experiments show that, compared with schemes that only employ static draft length or fixed threshold strategies, the E provided by this invention... 2 -SCI achieves a better trade-off between latency, throughput and accuracy on multiple task datasets: on the one hand, it effectively reduces the average generation latency and the worst response time for long-tail requests; on the other hand, it significantly improves the overall resource utilization efficiency of the edge-cloud system while ensuring or even slightly improving task accuracy.

[0105] Comparative results in conventional single-machine scenarios and edge-cloud collaborative scenarios show that the method of the present invention has a significant acceleration and performance improvement effect on other mainstream edge-cloud inference decoding methods.

[0106] To verify the effectiveness of the proposed distributed tolerance threshold, historical window information, and progressive pre-aiming concurrency (PLC) strategy, a series of ablation experiments were conducted on four datasets. The results show that, under most configurations, retaining the distributed tolerance threshold comparison module can significantly improve the token generation speed. Once historical information is removed, the performance degradation is most significant when only the static threshold sliding mechanism is used, which fully demonstrates that the historical window integration strategy proposed in this invention makes a key contribution to the overall performance.

[0107] The test results above show that the method proposed in this invention effectively balances latency and throughput requirements in an edge-cloud collaborative environment while maintaining the inference accuracy of large models. Compared with existing speculative decoding acceleration methods, it has more outstanding comprehensive performance and provides a technical path with both high efficiency and high reliability for the deployment of large models in resource-constrained scenarios.

[0108] This invention provides an electronic device, including: a computer-readable storage medium and a processor;

[0109] The computer-readable storage medium is used to store executable instructions;

[0110] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0111] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0112] This invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in any of the above embodiments.

[0113] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for accelerating edge-cloud collaborative speculative decoding based on credit inertia, characterized in that, include: S1, send the draft window consisting of the 1st to Wth tokens predicted by the draft model on the edge side as the current decoding window to the cloud side; at the same time, the draft model continues to make predictions; W>1; The input to the draft model is a sequence of numbers corresponding to the natural language text input by the user; S2, the target model on the cloud side verifies the first token in the current decoding window according to the verification threshold of the current decoding window. If the verification is successful, the target model verifies the other tokens in the current decoding window in sequence according to the verification threshold of the current decoding window. Otherwise, the target model generates a full-window rejection signal, discards all tokens in the current decoding window, resamples and generates a new token, and sends it along with the full-window rejection signal to the draft model. The draft model updates the context according to the new token, and the verification of the current decoding window ends. In this process, when the target model sequentially verifies other tokens in the current decoding window according to the verification threshold, if a token fails verification, it discards that token and the remaining tokens in the current decoding window, resamples and generates a new token, and sends it along with the sequence number of the verified tokens to the draft model. The draft model discards all tokens after the verified token and appends the resampled token to the end of the verified tokens, updates the context, and the verification of the current decoding window ends. If all tokens in the current decoding window pass verification, the target model sends the entire window's receive signal to the draft model, and the verification of the current decoding window ends. S3, sequentially take the draft window composed of the (W+1)th to (2W)th tokens predicted by the draft model, the draft window composed of the (2W+1)th to (3W)th tokens, ..., the draft window composed of the (AW+1)th to (A+1)×Wth tokens as the current decoding window and return to S2, until the termination condition is met; A>1; In step S2, if the current decoding window is in the [missing information]... The distribution divergence of each token If the value is less than or equal to the verification threshold of the current decoding window, the verification passes; otherwise, the verification fails. in, ; and Let be the probability distributions of the i-th token output by the draft model and the target model, respectively. It is a mixed distribution. , In mixed distribution Down of divergence, In mixed distribution Down of Divergence.

2. The method as described in claim 1, characterized in that, In step S2, before the target model on the cloud side verifies the first token in the decoding window based on the verification threshold of the current decoding window, the following steps are also included: (1) Calculate the instantaneous pass rate of the K historical decoding windows t-1, t-2, ..., tK adjacent to the current decoding window t. ;in, , For historical decoding window The pass rate, j = t-1, t-2, ..., tK, , For historical decoding window The Middle The distribution divergence of each token For historical decoding window The verification threshold; For indicator functions, hour, ,otherwise ; (2) Calculate the average pass rate of the K historical decoding windows t-1, t-2, ..., tK adjacent to the current decoding window t. ;in, = ; (3) According to Calculate the smoothing pass rate of the K historical decoding windows t-1, t-2, ..., tK adjacent to the current decoding window t. ;in, , These are the weighting coefficients. The smoothed pass rate of the K historical decoding windows t-2, t-3, ..., t-1-K adjacent to the previous decoding window t-1; (4) and The input is fed into a pre-trained predictor to obtain the validation threshold for the current decoding window t. Predicted value ,like and If the absolute value of the difference exceeds the set value, and the absolute value of the difference between the predicted value and the verification threshold of each of the P previous historical decoding windows adjacent to the current decoding window exceeds the set value, then... The value is updated to The value, otherwise The value remains unchanged.

3. The method as described in claim 2, characterized in that, The training process of the predictor includes: (1) Constructing the training dataset: Set the validation threshold of all decoding windows to a given threshold T, decode the natural language text samples input by the user, and calculate the validation threshold of each decoding window under different given threshold T configurations. The joint cost is used to select the N decoding windows with the smallest joint cost. For each of the N decoding windows, the smooth pass rate and instant pass rate of the K historical decoding windows adjacent to it, along with the corresponding given threshold, are used as the training dataset. Among them, each decoding window Joint cost under a given threshold T configuration ; express End-to-end latency, including draft model generation, under a given threshold T configuration. Time and target model validation The time required for the verification results to be returned to the draft model; express The pass rate under a given threshold T configuration; For the weighting factor; (2) The predictor is trained with the smooth pass rate and instant pass rate of the K historical decoding windows adjacent to the decoding window in the training dataset as inputs, and with the goal of minimizing the difference between the predicted value of the verification threshold of the decoding window output by the predictor and the given threshold; wherein the predictor is a neural network.

4. An electronic device, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any one of claims 1-3.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to perform the method as described in any one of claims 1-3.

6. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-3.

Images