End-cloud collaborative speculative decoding acceleration method based on credit inertia

Abstract

The invention discloses an end-cloud collaborative speculative decoding acceleration method based on credit inertia, which belongs to the field of edge-cloud collaborative computing, and improves the efficiency and stability of end-cloud collaborative generation through credit inertia perception, adaptive threshold prediction and asynchronous parallel scheduling. The method comprises the following steps: firstly, constructing a steady distribution level verification criterion by using a distribution difference between a Jensen-Shannon divergence measurement draft model and a cloud target model; then, extracting a credit inertia characteristic of a historical window through exponential weighted moving average, and adaptively adjusting a receiving threshold value and a draft window length through a lightweight threshold value prediction network; and finally, a progressive preview concurrent mechanism is introduced, draft generation, cloud pre-auditing and full auditing are decoupled into an asynchronous assembly line, so that the end-to-end time delay is remarkably reduced on the premise of ensuring the generation quality, invalid rollback is reduced, the effective token throughput rate is improved, and the method is suitable for high-real-time applications such as mobile photo albums, multi-modal retrieval and description generation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of edge-cloud collaborative computing, and more particularly, relates to an edge-cloud collaborative speculative decoding acceleration method based on credit inertia. BACKGROUND

[0002] With the wide deployment of large models in intelligent customer service, code generation, traffic scheduling and other key businesses, interactive applications for end users are rapidly moving from "cloud centralization" to "edge-cloud collaboration". In this process, the latency, energy consumption and cost of large language model inference directly affect the business experience and system scalability. Due to the large parameter size and computationally intensive nature of cloud-side large models, relying solely on cloud-side inference often fails to meet the real-time response requirements under a large number of concurrent requests; while sinking the complete model to the edge or mobile device is subject to multiple constraints of computing power, storage and power consumption. Therefore, how to achieve low-latency and high-precision generative services in an edge-cloud collaborative framework through collaborative inference of lightweight small models and cloud-side large models has become a key problem in current intelligent application infrastructure.

[0003] Among the many acceleration techniques, speculative decoding provides a new acceleration path for large model inference at the system level by introducing a "draft-verification" dual-model structure: a small model on the edge or near the edge generates a candidate token sequence first, and then a large model on the cloud verifies and corrects it, thereby improving decoding throughput without significantly sacrificing accuracy. However, existing speculative decoding schemes are mostly designed for single-machine or homogeneous cluster scenarios, with the implicit assumption of relatively abundant and stable computing and communication resources. When this paradigm is migrated to the actual edge-cloud collaborative environment, factors such as limited uplink bandwidth, significant network latency fluctuations, and highly heterogeneous edge-side computing power can amplify the risks caused by the distribution bias between the draft model and the target model: once the draft sequence is rolled back on a large scale, not only is the expected acceleration benefit lost, but additional recalculation and communication overhead is introduced, resulting in an increase in end-to-end latency.

[0004] On the other hand, existing methods usually rely on pre-set fixed acceptance thresholds or simplified statistical assumptions to determine whether the draft token can be "accepted" by the target model. Such static thresholds exhibit obvious rigidity problems under real business traffic: when the input context, task type or network condition changes, overly conservative thresholds will frequently reject high-quality drafts that could have been accepted, reducing effective throughput; while overly aggressive thresholds will lead to error accumulation and large-scale rollback, affecting generation quality. In continuous dialogue, multi-round interaction and long text generation scenarios, the pattern of draft acceptance or rejection often has obvious time correlation, but this kind of cross-window statistical information and "inertia" characteristics have not been systematically exploited and utilized, making it difficult for edge-cloud collaborative speculative decoding to adaptively adjust strategies based on historical performance.

[0005] Meanwhile, from the system perspective, the existing end-cloud collaborative inference framework often has a clear "mutual waiting" phenomenon between draft generation and cloud verification: the end side needs to wait for the verification results of the previous batch of tokens before safely advancing the context, and the cloud side relies on the end side to upload the complete draft sequence to start efficient parallel verification; superimposed with unstable network transmission delay, it is easy to form a performance bottleneck in the inference pipeline. The lack of joint modeling and dynamic scheduling of end-side computing, cloud-side load, and network conditions also makes it difficult for the system to balance response time, cloud energy consumption, and service quality in different business scenarios. SUMMARY

[0006] In view of the above defects or improvement needs of the prior art, the present application provides an end-cloud collaborative speculative decoding acceleration method based on credit inertia, thereby solving the performance bottleneck caused by the fixed threshold rigidity, end-cloud mutual waiting, and network delay fluctuation in traditional end-cloud speculative decoding.

[0007] To achieve the above-mentioned purpose, according to the first aspect of the present application, an end-cloud collaborative speculative decoding acceleration method based on credit inertia is provided, comprising: S1, the draft window composed of the first-W token predicted and output by the draft model on the edge side is sent to the cloud side as the current decoding window; at the same time, the draft model continues to make prediction output; W>1; Wherein, the input of the draft model is a digital sequence corresponding to the natural language text input by the user; S2, the target model on the cloud side verifies the first token in the current decoding window according to the verification threshold of the current decoding window, if the verification is passed, the target model verifies the other tokens in the current decoding window in sequence according to the verification threshold of the current decoding window, otherwise the target model generates an entire window rejection signal, discards all tokens in the current decoding window, generates a new token by resampling, sends it and the entire window rejection signal to the draft model, and the draft model updates the context according to the new token, and the current decoding window verification ends; If a token that fails to pass the verification occurs, the target model discards the token and the remaining tokens in the current decoding window, generates a new token by resampling, sends the new token and the serial number of the token that has passed the verification to the draft model, and discards all tokens after the token that has passed the verification by the draft model, and adds the resampled token to the end of the token that has passed the verification, updates the context, and ends the verification of the current decoding window. S3, respectively, sequentially take the draft window composed of the (W+1)th to 2Wth tokens, the draft window composed of the (2W+1)th to 3Wth tokens, …, and the draft window composed of the (AW+1)th to (A+1)×Wth tokens predicted by the draft model as the current decoding window and return to S2 until a termination condition is met; A>1.

[0008] According to a second aspect of the present application, an electronic device is provided, comprising: a computer readable storage medium and a processor. The computer readable storage medium is configured to store executable instructions. The processor is configured to read the executable instructions stored in the computer readable storage medium and execute the method according to the first aspect.

[0009] According to a third aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for causing a processor to execute the method according to the first aspect.

[0010] According to a fourth aspect of the present application, a computer program product is provided, comprising computer programs or instructions, which, when executed by a processor, implement the method according to the first aspect.

[0011] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects compared with the prior art: The end-cloud collaborative speculative decoding acceleration method based on credit inertia provided by the present application is used for end-cloud collaborative acceleration reasoning of natural language generation tasks of text data, comprising: collecting user inputted text to be processed, encoding the text to be processed into a token sequence, and inputting the token sequence into an end-cloud collaborative reasoning system composed of a pre-trained edge-side lightweight draft model and a cloud-side target large model, using adaptive threshold control and progressive parallel decoding mechanism driven by credit inertia to segment, speculate and verify the generation process, and finally output coherent and reliable generated text results; the method has the following advantages: 1. The method provided by the present application decouples the generation of the draft and the verification of the cloud, synchronously depends on the verification process of the cloud, divides the verification process of the cloud into two asynchronous stages of pre-trial and full trial, in the pre-trial stage, the target model of the cloud only verifies the first token in the current decoding window, if the verification fails, the entire window is immediately rejected and a re-generation signal is sent to the draft model to avoid invalid transmission, if the verification passes, the full trial stage is entered, the target model of the cloud asynchronously verifies the remaining tokens in the current decoding window, while the draft model continues to generate the next draft window, realizing the time overlap execution of the two sides of the end-to-cloud, which can reduce the end-to-end delay and improve the resource utilization.

[0012] 2. As a further preferred scheme, the method provided by the present application converts the discrete token verification into a continuous distribution measurement task, uses the Jensen-Shannon divergence to characterize the symmetric difference between the output distribution of the draft model and the output distribution of the target model, which can further improve the decoding quality.

[0013] 3. As a further preferred scheme, the method provided by the present application considers that the speculative decoding based on the fixed verification threshold is easy to cause the high-quality draft token to be excessively rejected, adopts an adaptive verification threshold switching mode based on credit inertia, uses the smooth pass rate of the K historical decoding windows adjacent to the current decoding window as a credit score, predicts the verification threshold of the current decoding window according to the credit score and the instant pass rate of the K historical decoding windows adjacent to the current decoding window, and simultaneously adopts a hysteresis control strategy: only when the difference between the predicted value and the current value of the verification threshold exceeds a set value and has lasted for P historical decoding windows, the update is triggered, avoiding the performance fluctuation caused by frequent oscillation. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 The flowchart of the end-to-cloud multi-modal adaptive speculative decoding method based on credit inertia provided by the embodiment of the present application.

[0015] Figure 2 The training flowchart of the threshold predictor provided by the embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0017] In an edge-cloud collaborative large model inference scene, edge-cloud mutual waiting and network latency fluctuation cause high inference delay and insufficient throughput. Based on this, an embodiment of the present application provides an edge-cloud collaborative speculative decoding acceleration method based on credit inertia, which is used for edge-cloud collaborative acceleration inference of natural language generation task of text data. The user input can be a dialogue query, a code completion request or a document summary instruction. The method comprises: S1, the draft window composed of the first-W tokens of the draft model prediction output on the edge side (i.e. the first draft window) is sent to the cloud side as the current decoding window; at the same time, the draft model continues to perform prediction output; W>1; Wherein, the input of the draft model is a digital sequence corresponding to the natural language text input by the user; S2, the target model on the cloud side verifies the first token in the current decoding window according to the verification threshold of the current decoding window. If the verification is passed, the target model verifies the other tokens in the current decoding window in order according to the verification threshold of the current decoding window. Otherwise, the target model generates an entire window rejection signal, discards all tokens in the current decoding window, and generates a new token by resampling, and sends it to the draft model together with the entire window rejection signal. The draft model updates the context according to the new token, and the current decoding window verification ends; Wherein, when the target model verifies the other tokens in the current decoding window in order according to the verification threshold of the current decoding window, if a token that does not pass the verification appears, the token and the remaining tokens in the current decoding window are discarded, a new token is generated by resampling, and it is sent to the draft model together with the serial number of the token that has passed the verification. The draft model discards all tokens after the token that has passed the verification, and adds the resampled token to the end of the token that has passed the verification, updates the context, and the current decoding window verification ends. If all tokens in the current decoding window pass the verification, the target model sends an entire window acceptance signal to the draft model, and the current decoding window verification ends; S3, the draft window composed of the W+1-2W tokens of the draft model prediction output (i.e. the second draft window), the draft window composed of the 2W+1-3W tokens (i.e. the third draft window), …, the draft window composed of the AW+1-(A+1)×W tokens (i.e. the A-th draft window) are returned to S2 as the current decoding window in turn, until the termination condition is met; A>1.

[0018] In step S1, first, the natural language text input by the user is obtained, and after being coded by a tokenizer (the length of the coded token sequence is variable), it is mapped to a corresponding numbered list, i.e., a digital sequence. The digital sequence is input to the edge-side lightweight draft model as context, and autoregressive prediction is performed to generate a token sequence. The draft model simultaneously outputs a probability distribution of each token, which is used for subsequent distribution measurement and credit evaluation.

[0019] The end-cloud collaborative system supports two working modes of batch processing and stream generation.

[0020] The method provided by the application decouples and synchronously depends on the draft generation and cloud verification to minimize the delay. In the traditional method, the draft model needs to wait for the verification feedback, and the target model needs to wait for the complete draft sequence, which causes the uplink transmission delay and the calculation waiting time to be superimposed. Therefore, in step S2, the method provided by the application adopts a progressive lookahead concurrency (PLC) mechanism to divide the verification process in the cloud into two asynchronous stages of pre-trial and full trial: in the pre-trial stage, the target model in the cloud only verifies the first token in the current decoding window t If the verification fails, the entire window is immediately rejected, and a re-generation signal is sent to the draft model , and the target model resamples a new token and sends it to the draft model for whole-window rollback to avoid invalid transmission; if the verification passes, the full-trial stage is entered, and the target model in the cloud asynchronously verifies the remaining tokens in the current decoding window , and the draft model continues to generate the next draft window.

[0021] The above process maximizes the throughput by time overlap:

[0022] wherein, and Through pipeline parallelism and concealment, only the communication overhead and the re-generation time after rejection are reserved.

[0023] To ensure the consistency of the KV cache, the draft model maintains a double buffer: the main buffer stores the verified prefix, and the shadow buffer stores the speculative continuation. When the draft model receives a rejection signal, it discards the shadow buffer and recovers the state from the main buffer; when it receives an acceptance signal, it merges the two buffer areas and updates the main buffer pointer. By constructing such a complete asynchronous collaborative mechanism, the method integrates the dispersed and serial verification process into a compact and parallel inference pipeline, thereby constructing an efficient and low-delay end-cloud collaborative closed loop.

[0024] Specifically, asFigure 1 To further reduce end-to-end latency and improve resource utilization, the application decouples draft generation and cloud verification into two stages of pre-trial and full trial based on the Progressive Lookahead Concurrency (PLC) mechanism, realizing time overlap execution on the edge and cloud sides. The two stages are further described below.

[0025] (1) Pre-trial stage: First token quick screening In the PLC mechanism, when the edge-side draft model generates a new draft window of length K, it immediately sends the window as the current decoding window t and the corresponding distribution information to the cloud. The target model on the cloud first verifies the first token of the current decoding window t. The cloud calculates the distribution divergence of the first token and compares it with the verification threshold If the distribution divergence of the first token is greater than , it is considered that the overall quality of the window is poor, and the cloud does not need to continue to verify the remaining tokens, and immediately generates a rejection signal to return to the edge side. If the distribution divergence of the first token is less than or equal to , it is considered that the window has high potential for passing, and the remaining tokens in the current decoding window t are fully verified in the full trial stage.

[0026] Through this "first token pre-trial" strategy, a large number of draft windows generated by the edge-side draft model with poor quality can be eliminated in a very short time, avoiding invalid calculation and transmission of the remaining tokens.

[0027] It can be understood that when the first draft window generated by the draft model is verified, t = 1, when the second draft window generated by the draft model is verified, t = 2, and so on.

[0028] (2) Full trial stage: Asynchronous verification and draft lookahead generation When the current decoding window t passes the pre-trial, the cloud starts the full trial process to measure the distribution and accept the remaining tokens in the window. During this period, the edge-side draft model no longer waits for the verification result of the current decoding window, but continues to generate the next draft window based on the tentative context, forming a forward-looking draft sequence.

[0029] In this process, the system maintains two sets of KV caches, main buffer and shadow buffer: ​The main buffer stores the token prefix that has been finally confirmed by the cloud (i.e., the draft window that has passed the verification of the cloud); the shadow buffer stores the speculative continuation writing based on the window that has not been confirmed. When the cloud completes the full review of the current window, if the window is accepted, an acceptance signal is returned, the edge side merges the shadow buffer content into the main buffer, and updates the context; if the window is rejected, the corresponding token in the shadow buffer is discarded according to the rejection signal, a token generated by the target model is received and merged into the end of the token sequence that has passed the verification, and the draft model continues to generate subsequent tokens according to the sequence. Through this double buffering mechanism, the system maximizes the parallelism of draft generation and cloud verification under the premise of ensuring the consistency of the context.

[0030] It can be understood that when the first draft window generated by the draft model is verified, t = 1, when the second draft window generated by the draft model is verified, t = 2, and so on.

[0031] For example, assuming that the length of each draft window is W = 5 (i.e., each draft window consists of 5 tokens), when the second draft window generated by the draft model (consisting of the 6th to 10th tokens generated by the draft model) is verified: Assuming that the first token of the second draft window (i.e., the 6th token generated by the draft model) does not pass the verification, the target model generates an entire window rejection signal and sends it to the cloud, discards all tokens in the second draft window, and generates a new token on the basis of the verified context (i.e., the 1st to 5th tokens) and sends it to the edge side. The draft model of the edge side updates the context in the shadow buffer: the draft model discards all tokens in the second draft window (i.e., the 6th to 10th tokens originally generated by the draft model) and all tokens in all draft windows after the second draft window in the shadow buffer, takes the new token as a new 6th token, and regenerates subsequent tokens; Assuming that the first token and the second token (i.e. the 6th-7th token generated by the draft model) in the second draft window pass the verification, the third token (i.e. the 8th token generated by the draft model) in the second draft window fails the verification, the target model generation part sends a rollback signal to the edge end, and a new token is generated by resampling on the basis of the verified context (i.e. the 1st-7th token), which is sent to the draft model together with the serial number (i.e. 1-2) of the verified token in the second draft window, and the draft model discards the 3rd-5th token in the second draft window according to the signal, and updates the context: after the draft model discards all tokens after the verified token (i.e. the 1st-7th token), the new token sent by the cloud is taken as the 8th token, and the subsequent draft token is generated on the basis of the existing one.

[0032] Although the cloud verification can guarantee the generation quality, the traditional method mostly uses the confidence ratio of the draft token and the target token for negative sampling. However, the traditional negative sampling strategy based on the confidence ratio only focuses on the single token probability and ignores the global consistency of the draft-target distribution, resulting in that the token with local high quality but global deviation is wrongly accepted. To solve this problem, the method provided by the present application converts the discrete token verification into a continuous distribution measurement task, uses the Jensen-Shannon divergence to characterize the symmetric difference between the output distribution of the draft model and the output distribution of the target model, and establishes a more robust verification criterion. The process is realized by a distribution measurement function , that is, as a further preferred solution, in step S2, if the distribution divergence of the first token in the current decoding window is less than or equal to the verification threshold value of the current decoding window, the verification passes, otherwise the verification fails; wherein the distribution divergence of the first token in the current decoding window is calculated according to the following formula: ; wherein, and are the probability distributions of the i-th token output by the draft model and the target model respectively, is a mixed distribution, , is the Jensen-Shannon divergence under the mixed distribution , ​​​​of divergence; is and a distribution similarity calculation function, is and JS divergence.

[0033] This way effectively realizes the upgrade from point estimation to distribution estimation.

[0034] The method provided by the application can set the verification threshold of each current decoding window to the same fixed threshold, and verify the token of each current decoding window according to the fixed threshold in step S2.

[0035] Considering that the traditional method mostly uses a fixed threshold to verify the draft / target token distribution similarity, high-quality tokens are often excessively rejected due to the neglect of time sequence dependence, and it is difficult to apply in mobile terminal scene applications with strict real-time requirements. After research, it is found that when the window pass rate of the end-to-cloud speculative decoding process is counted, on the MMLU and QuALITY data sets, the token pass rate of the continuous window is not randomly fluctuating, but presents a pattern of maintaining a "high pass rate interval" or a "low pass rate interval" for a long time, and state jumping only occurs at a few positions. This shows that the performance of the current window is highly related to the previous window, and there is obvious "credit inertia": once in a high credit state, the following multiple windows tend to be "good->good"; and when in a low credit state, it is easy to be continuously "bad->bad". Based on this, as a further preferred scheme, the method provided by the application compresses the pass rates of the K adjacent historical decoding windows (i.e. the K verified decoding windows adjacent to the current decoding window) of the current decoding window into credit scores, and adaptively adjusts the verification threshold of the current decoding window t according to the high and low of the credit scores : appropriately relax the verification threshold in the high credit stage to improve the throughput, and tighten the verification threshold in the low credit stage to suppress invalid drafts and rollback, thereby forming an end-to-cloud speculative decoding strategy in which the verification threshold of the decoding window is dynamically adjusted over time. That is, preferably, in step S2, the target model on the cloud side verifies the first token in the decoding window according to the verification threshold of the current decoding window, and further includes: (1) calculating the instantaneous pass rate of the K adjacent historical draft windows (i.e. the K verified decoding windows adjacent to the current decoding window t) t-1, t-2,..., t-K of the current decoding window t ; wherein , , are the pass rates of the K adjacent historical decoding windows t-1, t-2, t-K of the current decoding window t; for convenience of description, To provide a unified expression For historical decoding window The pass rate, j = t-1, t-2, ..., tK, , For historical decoding window The Middle The distribution divergence of each token For historical decoding window The verification threshold; For indicator functions, hour, ,otherwise ; (2) Calculate the average pass rate of the K historical decoding windows t-1, t-2, ..., tK adjacent to the current decoding window t. ; in, = ; (3) According to Calculate the smoothing pass rate of the K historical decoding windows t-1, t-2, ..., tK adjacent to the current decoding window t. ; in, , These are the weighting coefficients. The smoothed pass rate of the K historical decoding windows t-2, t-3, ..., t-1-K adjacent to the previous decoding window t-1; (4) and The input is fed into a pre-trained predictor to obtain the validation threshold for the current decoding window t. Predicted value ,like and If the absolute value of the difference exceeds the set value, and the absolute value of the difference between the predicted value and the verification threshold of each of the P previous historical decoding windows adjacent to the current decoding window exceeds the set value, then... The value is updated to The value, otherwise The value remains unchanged.

[0036] In particular, in the practice of end-to-cloud speculation decoding, the token passing rate of adjacent decoding windows often presents strong time correlation, that is, the recent passing rate of a draft window is high (for example, greater than 0.8), and the subsequent performance is high with high probability, and the recent passing rate of a draft window is low (for example, less than 0.4), and the performance is poor. This "credit inertia" is a key clue for adaptive verification, but the traditional method lacks systematic modeling of historical patterns. Considering that fixed threshold speculation decoding is prone to over-rejection of high-quality draft tokens, based on this, the method provided by the present application adopts a credit inertia-based adaptive threshold switching method: when the token passing rate of the recent historical window draft is high, the verification threshold is appropriately relaxed, and when the token passing rate of the recent historical window draft is low, the verification threshold is more stringent, thereby reducing the rejection rate. Exponentially weighted moving average (EWMA) is used to capture the time dynamics and generate a smooth passing rate estimate: First, the average passing rate of the K historical decoding windows adjacent to the current decoding window (that is, the K verified decoding windows adjacent to the current decoding window) is calculated :

[0037] Wherein, is the distribution divergence of the i-th token in the j-th historical decoding window, and W is the size of the draft window (that is, the size of the current decoding window and the historical decoding window, both of which are W), is an indicator function, , , otherwise .

[0038] Then, in order to avoid the influence of single window fluctuation on the strategy, an exponentially weighted moving average mechanism is used to smooth the average passing rate of the K historical decoding windows to the credit inertia estimate That is, based on , the EWMA is used to calculate the smoothed passing rate of the K historical decoding windows adjacent to the current decoding window t :

[0039] Wherein, is a weight coefficient, the value range is (0, 1), and is used to balance short-term responsiveness and long-term stability. The smoothed passing rate of the current decoding window t is a compressed representation (that is, a credit score) of the instantaneous passing rate of the K historical decoding windows t-1, t-2, …, t-K adjacent to the current decoding window , and correspondingly, is the passing rate of the historical decoding window t-1, is the passing rate of the historical decoding window t-2, the smoothed pass rate of the K historical decoding windows adjacent to the current decoding window t .

[0040] In this way, the responsiveness to short-term fluctuations and the stability of long-term trends are balanced, discrete validation events are converted into continuous state signals, and high-quality input features are provided for threshold prediction.

[0041] Finally, the smoothed pass rate of the K historical decoding windows adjacent to the current decoding window t is input to the pre-trained predictor together with the instantaneous pass rate to obtain the predicted value of the validation threshold of the current decoding window t If the absolute value of the difference between the predicted value and the current value of the validation threshold exceeds a set value, and the predicted values of the validation thresholds of the previous P historical decoding windows adjacent to the current decoding window each correspond to the validation thresholds of the previous P validated decoding windows, the value of the validation threshold of the current decoding window is updated to the value of the validation threshold of the previous decoding window, otherwise the value of the validation threshold of the current decoding window remains unchanged.

[0042] The method provided by the present application adopts an online stability guarantee mechanism and introduces a hysteresis control strategy: only when the difference between the predicted value and the current value of the validation threshold exceeds a set value and has lasted for P historical decoding windows, the update is triggered, avoiding performance fluctuations caused by frequent oscillations. By using the two parameters and P, the method provided by the present application can greatly reduce the disturbance to the end-to-cloud pipeline caused by frequent adjustments while ensuring adaptability, thereby improving the stability and predictability of the overall inference process.

[0043] For example, assuming P=3, for the current decoding window t, when the absolute value of the difference between the predicted value and the current value of the validation threshold exceeds a set value, the value of the validation threshold of the current decoding window t is only updated to the value of the validation threshold of the previous decoding window if the absolute value of the difference between the predicted value and the current value of the validation threshold of each of the historical decoding windows t-1, t-2, and t-3 exceeds the set value, otherwise the value of the validation threshold of the current decoding window t remains unchanged. It can be understood that for the first to Kth draft windows of the draft model output, the validation thresholds thereof are all initial values when decoding, and the judgment of whether to adaptively update the validation thresholds is involved only when the K+1th draft window is decoded.

[0044]

[0045] ​​​​​​​To address the computational overhead of real-time calculation of the optimal threshold in online inference, the method provided in this invention employs a data-driven mapping learning strategy. For example... Figure 2 As shown, a nonlinear mapping from the instantaneous pass rate and smooth pass rate of the K historical decoding windows adjacent to the decoding window to the optimal verification threshold of the decoding window is constructed through offline calibration, and a neural network-based predictor (also known as a threshold predictor) is trained to achieve fast prediction of the verification threshold.

[0046] The training process of the predictor includes: (1) Constructing the training dataset: Set the validation threshold of all decoding windows to a given threshold T, decode the natural language text samples input by the user, and calculate the validation threshold of each decoding window under different given threshold T configurations. The joint cost is used to select the N decoding windows with the smallest joint cost. For each of the N decoding windows, the smooth pass rate and instant pass rate of the K historical decoding windows adjacent to it, along with the corresponding given threshold, are used as the training dataset. Among them, each decoding window Joint cost under a given threshold T configuration ; express End-to-end latency, including draft model generation, under a given threshold T configuration. Time and target model validation The time required for the verification results to be returned to the draft model; express The pass rate under a given threshold T configuration; This is a weighting factor.

[0047] (2) The predictor is trained with the smooth pass rate and instant pass rate of the K historical decoding windows adjacent to the decoding window in the training dataset as inputs, and with the goal of minimizing the difference between the predicted value and the true value (i.e. the given threshold corresponding to the decoding window in the training set) of the decoding window output by the predictor; wherein the predictor is a neural network, for example, a five-layer feedforward network structure can be adopted.

[0048] The predictor is trained using an "offline calibration + online inference" approach, by constructing a training dataset. Complete the learning of the nonlinear mapping so that it can be based on the smooth pass rate of the K historical decoding windows adjacent to the current decoding window t. With real-time pass rate The optimal value of the prediction validation threshold is determined. Specifically, the training process of the threshold predictor is as follows: 1) Run log collection First, a complete edge-cloud speculative decoding framework was deployed in a representative application scenario and repeatedly run under different network bandwidths, latency, and device types to collect a large number of real requests. The edge-cloud time-overlapping decoding method based on a progressive pre-aiming concurrency mechanism, provided by this invention, was used to decode user-input natural language text samples, processing each decoding window for each request. Record the following information: the given value T of the dynamic threshold, and the smoothed pass rate of the K historical windows adjacent to the decoding window. With real-time pass rate Then calculate each decoding window The combined cost. Continuously change the value of T and repeat the above process.

[0049] In other words, when constructing the training dataset, a balance between end-to-end latency and token pass rate is considered, and a joint cost function is constructed. Used to calculate across all decoding windows When decoding user-input natural language text samples using the edge-cloud time-overlapping decoding method based on progressive pre-aiming concurrency mechanism provided by this invention, with all verification thresholds set to a given threshold T, each decoding window... The joint cost, while recording each decoding window real-time pass rate and smooth pass rate The given threshold T is continuously changed, and the joint cost of each decoding window is recalculated. Using this method, the joint cost of each decoding window under different values ​​of the given threshold T is tested offline in a limited number of natural language question-answering samples. Then, the N decoding windows with the lowest joint cost are selected. For each of these N decoding windows, the smooth pass rate and instant pass rate of its K neighboring historical decoding windows, along with the corresponding given threshold, are used as the training dataset. .

[0050] For example, suppose T uses 5 different values, from which 20 decoding windows with the minimum joint cost are selected. Then, for each of these 20 decoding windows, the smooth pass rate and instant pass rate of its K neighboring historical decoding windows, along with the corresponding given thresholds, are used as the training dataset. .

[0051] The optimal control strategies are summarized under different credit inertia levels, different network conditions, and different task difficulties.

[0052] 2) Predictor Structure Design and Training Threshold predictor ( (For predictor parameters) A multi-layer feedforward neural network structure is preferred.

[0053] When training the predictor, first select a decoding window, input the smooth pass rate and instant pass rate of the K adjacent historical decoding windows into the predictor, and record the predicted value of the validation threshold output by the predictor. Calculate the difference between this predicted value and the given threshold corresponding to the decoding window in the training set. Repeat this process until every decoding window in the training set is traversed. Update the predictor parameters through backpropagation to complete the training of the predictor.

[0054] After training is complete, the trained predictor will be... It is permanently deployed in the edge-cloud control module.

[0055] During the online inference phase, the predictor The input is the smoothed pass rate of the K historical decoding windows adjacent to the current decoding window t. and real-time pass rate The output is the verification threshold for the current decoding window t. Predicted value , that is:

[0056] The process of generating offline training data is as follows: exhaustively search on representative datasets. The parameters are combined, and their joint cost is evaluated on the validation set for each configuration. The configuration that minimizes the objective function is selected as the supervision label, and the training dataset is constructed. ; To maximize inference efficiency, the entire verification process is sensitive to system performance bottlenecks. Finally, all intermediate statistics generated in the first three stages of the credit-inertia-based edge-cloud multimodal adaptive inference decoding framework—from draft to target distribution measurement to the final threshold decision—are seamlessly integrated into a standardized, real-time-scheduling verification strategy. This is implemented through a lightweight predictor that takes all historical verification statistics as input and instructs the neural network to generate a configuration according to a predefined joint optimization objective. The strategy structure is rigorous, including key modules such as request overview, distribution divergence sequence, verification starting point (credit inertia state awareness), adaptive adjustment path (dynamic updating and evaluation of the threshold, highlighting historical evidence of the optimal window size), final pass rate and latency profile, and asynchronous scheduling suggestions and optimization points. By generating such a complete configuration, this method materializes the complex, distributed verification process into a dynamic, traceable policy record, thereby constructing an efficient, adaptive inference closed loop.

[0057] In summary, the application provides an end-cloud collaborative inference decoding acceleration method based on credit inertia for end-cloud collaborative generation of text or multi-modal data, which comprises: encoding the user input and sending it to the edge-side lightweight draft model to generate a token sequence, constructing a current decoding window and sending it to the cloud; in the cloud target model, calculating the divergence of the output distribution of the token in the current decoding window according to the distribution metric function, and combining the immediate pass rate of the K historical decoding windows adjacent to the current decoding window to obtain the credit inertia estimation (i.e. smoothed pass rate); taking it as input, outputting the predicted value of the validation threshold of the current decoding window through the threshold predictor and judging whether to update the validation threshold; judging whether the token is accepted based on the validation threshold, generating the corresponding acceptance or rollback signal, and scheduling the draft generation and cloud verification through the progressive end-cloud asynchronous flow mechanism; after the decoding satisfies the termination condition, the token sequence finally confirmed by the cloud is decoded into natural language text or description result as the final output.

[0058] The method adopts asynchronous verification, distribution metric verification based on JS divergence, threshold prediction based on historical statistics and other strategies, and can systematically guide the lightweight draft model and the cloud target model to collaboratively generate high-quality descriptions. The method mines the pass rate correlation of the draft window in the time sequence, regards it as a "credit signal" in the end-cloud collaborative inference process, and accordingly adjusts the decoding threshold, draft length and end-cloud flow concurrency strategy, so as to significantly reduce the computational and communication overhead caused by invalid verification and rollback under the premise of ensuring generation quality, and provides an extensible technical foundation for efficient deployment of large models in a heterogeneous end-cloud environment; a dynamic credit score is constructed between the end-side lightweight draft model and the cloud verification model using the historical window pass rate, and the verification threshold is adaptively adjusted to realize joint constraint of draft quality and end-cloud communication overhead. At the system level, a progressive end-cloud parallel pipeline is designed to decouple the draft generation, segmented verification and feedback submission process, and reduce invalid rollback and blocking waiting through asynchronous queues and hardware-aware scheduling. A large number of experiments show that the method provided by the application significantly reduces the end-to-end latency and improves the effective throughput under the premise of ensuring the generation quality, and provides an efficient and scalable solution for speculative decoding acceleration of large models in an end-cloud collaborative environment.

[0059] The end-cloud collaborative speculative decoding method (Elastic Edge-Cloud Speculative Decoding, E 2A systematic evaluation was conducted on the GSM8K mathematical reasoning dataset, the CommonsenseQA commonsense question answering dataset, the MMLU large-scale multi-task general understanding dataset, and the HumanEval code generation dataset, which were selected in terms of reasoning tasks to cover different types of load scenarios such as mathematical reasoning, factual question answering, comprehensive knowledge, and program generation. Various mainstream large language models were used, such as the Qwen2 series, Llama 3.1 series, Gemma2 series, and DeepSeek-R1-Distill-Qwen 1.5B+32B, to simulate different sizes of "edge small model + cloud large model" combinations to verify the universality and stability of the method under different model architectures and parameter sizes.

[0060] The experiment was conducted on a distributed end-to-cloud test platform composed of NVIDIA RTX 4090 and A6000 GPUs, which simulated a typical edge device and cloud server collaborative deployment scenario through a network environment with controllable bandwidth and latency. The batch size was set to 1 during inference to approximate the token-by-token generation mode under real online interactive services. The evaluation indicators included the number of tokens generated and finally accepted by the cloud per unit of time (throughput, tokens / s), the average inference latency of single tokens and the entire output, the acceleration ratio (relative to the standard beam search decoding baseline) under end-to-cloud collaboration, and the task accuracy indicators on each dataset (such as the accuracy of GSM8K, the accuracy of CommonsenseQA and MMLU, and the pass rate of HumanEval). The comparison results in the conventional single-site scenario and the end-to-cloud collaborative scenario showed that the E 2 Under the premise of maintaining or even slightly improving task accuracy, the E

[0061] Under various typical reasoning tasks and model combinations, compared with the traditional beam search decoding method without using the credit inertia mechanism, the E 2 The E 2The end-to-end latency of SCI is obviously lower than the baseline method, the number of tokens accepted by the cloud end per unit time (effective throughput) is steadily improved, and good robustness is shown under different network bandwidth and delay settings.

[0062] Further comparative experiments show that, compared with a scheme using only a static draft length or a fixed threshold strategy, the E 2 SCI achieves better "delay-throughput-accuracy" trade-offs on multiple task datasets: on the one hand, the average generation delay and the worst-case response time of long-tail requests are effectively reduced; on the other hand, the overall resource utilization efficiency of the end-cloud system is significantly improved while ensuring even slightly improved task accuracy.

[0063] The comparative results in the conventional single-site scenario and the end-cloud collaborative scenario show that the method of the present application has obvious acceleration and performance improvement effect on other mainstream end-cloud speculation decoding methods.

[0064] In order to verify the effectiveness of the proposed distribution tolerance threshold, history window information and gradual pre-lookahead (PLC) strategy, a series of ablation experiments were conducted on four datasets, and the results showed that, under most configurations, retaining the distribution tolerance threshold comparison module can significantly improve the token generation speed; once the history information is removed and only the static threshold sliding mechanism is used, the performance degradation is most obvious, fully demonstrating that the history window integration strategy proposed in the present application has a key contribution to the overall performance.

[0065] From the above test results, it can be seen that the method proposed in the present application effectively balances the latency and throughput demand in the end-cloud collaborative environment while maintaining the accuracy of large model inference, and has more outstanding comprehensive performance compared with existing speculation decoding acceleration methods, providing a technical path with high efficiency and high reliability for large model deployment in resource-constrained scenarios.

[0066] An electronic device is provided in an embodiment of the present application, comprising: a computer readable storage medium and a processor; The computer readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer readable storage medium, and execute the method as described in any of the above embodiments.

[0067] A computer readable storage medium is provided in an embodiment of the present application, and the computer readable storage medium stores computer instructions, and the computer instructions are used to make a processor execute the method as described in any of the above embodiments.

[0068] The embodiment of the present application provides a computer program product, including a computer program or instructions, which are executed by a processor to realize the method as described in any of the above embodiments.

[0069] Those skilled in the art can understand that the above description is only the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for accelerating edge-cloud collaborative speculative decoding based on credit inertia, characterized in that, include: S1, send the draft window consisting of the 1st to Wth tokens predicted by the draft model on the edge side as the current decoding window to the cloud side; at the same time, the draft model continues to make predictions; W>1; The input to the draft model is a sequence of numbers corresponding to the natural language text input by the user; S2, the target model on the cloud side verifies the first token in the current decoding window according to the verification threshold of the current decoding window. If the verification is successful, the target model verifies the other tokens in the current decoding window in sequence according to the verification threshold of the current decoding window. Otherwise, the target model generates a full-window rejection signal, discards all tokens in the current decoding window, resamples and generates a new token, and sends it along with the full-window rejection signal to the draft model. The draft model updates the context according to the new token, and the verification of the current decoding window ends. In this process, when the target model sequentially verifies other tokens in the current decoding window according to the verification threshold, if a token fails verification, it discards that token and the remaining tokens in the current decoding window, resamples and generates a new token, and sends it along with the sequence number of the verified tokens to the draft model. The draft model discards all tokens after the verified token and appends the resampled token to the end of the verified tokens, updates the context, and the verification of the current decoding window ends. If all tokens in the current decoding window pass verification, the target model sends the entire window's receive signal to the draft model, and the verification of the current decoding window ends. S3, sequentially take the draft window composed of the W+1~2Wth tokens predicted by the draft model, the draft window composed of the 2W+1~3Wth tokens, ..., the draft window composed of the AW+1~(A+1)×Wth tokens as the current decoding window and return to S2, until the termination condition is met; A>1.

2. The method as described in claim 1, characterized in that, In step S2, if the current decoding window is in the [missing information]... The distribution divergence of each token If the value is less than or equal to the verification threshold of the current decoding window, the verification passes; otherwise, the verification fails. in, ; and Let be the probability distributions of the i-th token output by the draft model and the target model, respectively. It is a mixed distribution. , In mixed distribution Down of divergence, In mixed distribution Down of Divergence.

3. The method as described in claim 2, characterized in that, In step S2, before the target model on the cloud side verifies the first token in the decoding window based on the verification threshold of the current decoding window, the following steps are also included: (1) Calculate the instantaneous pass rate of the K historical decoding windows t-1, t-2, ..., tK adjacent to the current decoding window t. ;in, , For historical decoding window The pass rate, j = t-1, t-2, ..., tK, , For historical decoding window The Middle The distribution divergence of each token For historical decoding window The verification threshold; For indicator functions, hour, ,otherwise ; (2) Calculate the average pass rate of the K historical decoding windows t-1, t-2, ..., tK adjacent to the current decoding window t. ;in, = ; (3) According to Calculate the smoothing pass rate of the K historical decoding windows t-1, t-2, ..., tK adjacent to the current decoding window t. ;in, , These are the weighting coefficients. The smoothed pass rate of the K historical decoding windows t-2, t-3, ..., t-1-K adjacent to the previous decoding window t-1; (4) and The input is fed into a pre-trained predictor to obtain the validation threshold for the current decoding window t. Predicted value ,like and If the absolute value of the difference exceeds the set value, and the absolute value of the difference between the predicted value and the verification threshold of each of the P previous historical decoding windows adjacent to the current decoding window exceeds the set value, then... The value is updated to The value, otherwise The value remains unchanged.

4. The method as described in claim 3, characterized in that, The training process of the predictor includes: (1) Constructing the training dataset: Set the validation threshold of all decoding windows to a given threshold T, decode the natural language text samples input by the user, and calculate the validation threshold of each decoding window under different given threshold T configurations. The joint cost is used to select the N decoding windows with the smallest joint cost. For each of the N decoding windows, the smooth pass rate and instant pass rate of the K historical decoding windows adjacent to it, along with the corresponding given threshold, are used as the training dataset. Among them, each decoding window Joint cost under a given threshold T configuration ; express End-to-end latency, including draft model generation, under a given threshold T configuration. Time and target model validation The time required for the verification results to be returned to the draft model; express The pass rate under a given threshold T configuration; For the weighting factor; (2) The predictor is trained with the smooth pass rate and instant pass rate of the K historical decoding windows adjacent to the decoding window in the training dataset as inputs, and with the goal of minimizing the difference between the predicted value of the verification threshold of the decoding window output by the predictor and the given threshold; wherein the predictor is a neural network.

5. An electronic device, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to perform the method as described in any one of claims 1-4.

7. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-4.

Images