Method, apparatus, device and storage medium for data processing

Abstract

The application discloses a data processing method, device, equipment and storage medium, and belongs to the technical field of computers. The method specifically comprises the following steps: obtaining a hardware resource usage and a draft history acceptance rate; wherein the hardware resource usage is determined based on the running condition of a GPU cluster system performing inference of a large language model, and the draft history acceptance rate is determined based on an inference result of the large language model; based on the hardware resource usage and the draft history acceptance rate, an inference acceleration strategy prediction model is used to obtain a speculation parameter adjustment strategy of a speculation decoding algorithm; and based on the speculation parameter adjustment strategy, the inference of the large language model is optimized.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to big data analysis technology, large model inference technology, and other technical fields, and particularly to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] Speculative decoding is a method to accelerate reasoning in large language models. It breaks down the traditional word-by-word recursive generation process into two stages: draft generation and parallel verification. A smaller and faster draft model first generates a sequence of candidate drafts (tokens) autoregressively, and then the original large model, i.e., the verification model, verifies these tokens in parallel.

[0003] Currently, speculative decoding solutions for related technologies include dual-model architectures and single-model inference with special structures. Although speculative decoding of related technologies can theoretically provide a speedup of 2 to 3 times or more, it still has many defects and limitations in practical large-scale distributed inference scenarios. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and storage medium, which can solve the problem of poor hyperparameter adjustment flexibility of speculative decoding algorithms in practical large-scale distributed inference scenarios. The technical solution is as follows:

[0005] Firstly, a data processing method is provided, the method comprising:

[0006] The hardware resource usage and draft history acceptance rate are obtained; wherein the hardware resource usage is determined based on the operation of the GPU cluster system that performs inference for the large language model, and the draft history acceptance rate is determined based on the inference results of the large language model.

[0007] Based on the hardware resource usage and the historical acceptance rate of the draft, the speculative parameter adjustment strategy of the speculative decoding algorithm is obtained by using the inference acceleration strategy prediction model.

[0008] Based on the aforementioned speculative parameter adjustment strategy, the inference of the large language model is optimized.

[0009] In one possible implementation, the inference acceleration strategy prediction model includes a hardware-aware cost model and an acceptance rate prediction model. The step of obtaining a speculative parameter adjustment strategy for the speculative decoding algorithm based on the hardware resource usage and the historical draft acceptance rate using the inference acceleration strategy prediction model includes:

[0010] Based on the hardware resource usage, hardware cost information is obtained using the hardware-aware cost model.

[0011] Based on the historical acceptance rate of the draft, the acceptance rate prediction model is used to obtain draft revenue information;

[0012] Based on the hardware cost information and draft profit information, a speculative parameter adjustment strategy for the speculative decoding algorithm is obtained.

[0013] In one possible implementation, obtaining the speculative parameter adjustment strategy for the speculative decoding algorithm based on the hardware cost information and draft profit information includes:

[0014] Based on the hardware cost information and the draft revenue information, the first revenue is determined;

[0015] Obtain the current second benefit;

[0016] Based on preset profit conditions, the first profit and the second profit, a speculative parameter adjustment strategy for the speculative decoding algorithm is obtained.

[0017] In one possible implementation, the step of obtaining the speculative parameter adjustment strategy for the speculative decoding algorithm based on the hardware resource usage and the historical acceptance rate of the draft, using an inference acceleration strategy prediction model, includes:

[0018] Based on the hardware resource usage, draft history acceptance rate, objective function, and constraints, the optimal solution of the inference acceleration strategy prediction model is determined.

[0019] Based on the optimal solution of the prediction model of the inference acceleration strategy, the speculative parameter adjustment strategy of the speculative decoding algorithm is obtained.

[0020] In one possible implementation, optimizing the inference of the large language model based on the speculative parameter adjustment strategy includes:

[0021] Based on the aforementioned speculative parameter adjustment strategy, speculative parameter information is obtained;

[0022] Based on the speculative parameter information, the inference of the large language model is optimized.

[0023] In one possible implementation, obtaining the speculative parameter information based on the speculative parameter adjustment strategy includes:

[0024] In response to the speculation parameter adjustment strategy of maintaining the current speculation parameter information, the current speculation parameter is obtained and used as the speculation parameter information;

[0025] In response to the speculation parameter adjustment strategy, which is to adjust the speculation parameter information, the adjusted speculation parameters are obtained and used as the speculation parameter information.

[0026] Secondly, a data processing apparatus is provided, the apparatus comprising:

[0027] An acquisition unit is used to acquire hardware resource usage and draft history acceptance rate; wherein, the hardware resource usage is determined based on the operation of the GPU cluster system executing inference of the large language model, and the draft history acceptance rate is determined based on the inference results of the large language model;

[0028] The prediction unit is used to obtain the speculative parameter adjustment strategy of the speculative decoding algorithm based on the hardware resource usage and the historical acceptance rate of the draft, using the inference acceleration strategy prediction model.

[0029] An optimization unit is used to optimize the inference of the large language model based on the strategy of adjusting the speculative parameters.

[0030] Thirdly, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the aspects and any possible implementations described above.

[0031] Fourthly, an electronic device is provided, comprising:

[0032] At least one processor; and

[0033] A memory communicatively connected to the at least one processor; wherein,

[0034] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described above and any possible implementations.

[0035] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the aspects and any possible implementations described above.

[0036] The beneficial effects of the technical solution provided in this application include at least the following:

[0037] As can be seen from the above technical solution, the embodiments of this application can obtain hardware resource usage and draft historical acceptance rate. The hardware resource usage is determined based on the operation of the GPU cluster system executing the inference of the large language model, and the draft historical acceptance rate is determined based on the inference result of the large language model. Then, based on the hardware resource usage and draft historical acceptance rate, the inference acceleration strategy prediction model can be used to obtain the speculative parameter adjustment strategy of the speculative decoding algorithm. Based on the speculative parameter adjustment strategy, the inference of the large language model can be optimized. Since the speculative parameter adjustment strategy of the speculative decoding algorithm can be analyzed and determined according to the hardware resource usage and the draft historical acceptance rate generated during the inference process of the large language model, the speculative parameters can be dynamically adjusted according to the real-time hardware operation, realizing adaptive adjustment of system resources and workload status, improving the flexibility of parameter adjustment, and thus ensuring the stability and consistency of the large model inference performance.

[0038] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic flowchart of a data processing method provided in one embodiment of this application;

[0041] Figure 2 This is a schematic diagram illustrating an application scenario of the architecture of the data processing method provided in one embodiment of this application;

[0042] Figure 3 This is a structural block diagram of a data processing apparatus provided in another embodiment of this application;

[0043] Figure 4 This is a block diagram of an electronic device used to implement the data processing method of the embodiments of this application. Detailed Implementation

[0044] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0045] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0046] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0047] Speculative decoding is a technique to accelerate Large Language Model (LLM) inference by breaking down the traditional word-by-word recursive generation process into two stages: "draft generation" and "parallel verification." A smaller, faster draft model (also known as an assistant model) first autoregressively generates a sequence of candidate tokens (draft sequences), which are then verified in parallel by the original large model (verification or target model). Through this two-stage process, the verification model can process multiple candidate tokens in a single forward propagation, thereby increasing parallelism and accelerating inference. The correctness of speculative decoding is guaranteed by the algorithm, for example, by using a rejection sampling strategy to ensure that the final generated sequence is consistent with the distribution directly generated by the original model. The key is that the candidate tokens generated by the draft model must have a high acceptance rate, i.e., the proportion of tokens the large model is willing to adopt; the higher the acceptance rate, the more significant the acceleration effect of speculative decoding.

[0048] Several speculative decoding implementations exist, including dual-model architectures (a small model and a large model) and single-model inference with special structures. For example, the Extrapolation Algorithm for Greater Language-model Efficiency (Eagle3) is a recent speculative decoding method. Eagle3 overlays a lightweight Transformer draft head onto the output of some layers of the original large model, enabling it to efficiently autoregressively generate multiple candidate tokens based on the current state of the large model. Because the Eagle series draft models are trained to align with the distribution of the large model, they are very compact while maintaining high acceptance rates, resulting in significant speedup. Eagle3 is considered the current performance benchmark for speculative decoding; its small model is trained in conjunction with the large model, allowing it to be used only with specific large models but with extremely high prediction accuracy. Another important approach is the TreeAttention mechanism, which allows the validation model to validate multiple branch sequences on a "candidate tree" in a single forward propagation. Because multiple sequences generated by the draft model often share a prefix before branching into a tree structure, a large model can perform parallel scoring on the entire candidate tree at once by constructing a special tree-like AttentionMask. This tree decoding method avoids the repeated calculation and storage of common prefixes, which can significantly reduce redundant calculations and key-value cache usage compared to sequence-by-sequence verification, thus improving the efficiency of the verification stage.

[0049] However, existing methods still have many shortcomings and limitations in real-world distributed inference scenarios involving large models: First, they lack resource awareness. Most current speculative decoding implementations do not adjust their strategies based on the real-time status of hardware resources. For example, when the system load is low and GPU computing power is abundant, a conservative speculative strategy may still be adopted, wasting available computing power; conversely, when the system load is high and close to saturation, continuing to speculate according to a fixed strategy may lead to unnecessary overhead. In high-load scenarios with limited system computing resources and a large number of requests, if a large amount of speculation is still performed, it may actually slow down the overall performance by wasting valuable computing power due to the computation of small models, rather than directly generating tokens per large model. Ideally, like the adaptive bitrate of video streams, the speculative intensity should be adjusted to high resolution during idle periods and low resolution during busy periods: bold speculation when resources are abundant, pushing more steps; converge speculation when resources are scarce, reducing overhead. Existing solutions lack such resource adaptability. Second, parameters are statically fixed. The key parameters of speculative decoding are usually statically set before system startup, or adjusted using only simple heuristic rules. For example, in some implementations, if all draft tokens pass validation in a round, the draft length is increased in the next round; otherwise, it is decreased. This static or pre-fixed parameter setting cannot adapt to changes in different tasks and contexts. Different request contents have vastly different requirements for the optimal draft length, and a fixed value often cannot satisfy both; in some cases, a fixed draft length can even be counterproductive. For instance, when the uncertainty of the output sequence is high, a longer draft length can lead to a sharp drop in acceptance rate, wasting computation; conversely, when the sequence trend is stable, a fixed short draft cannot fully utilize its potential. Furthermore, there is a mismatch between GPU memory and communication. In distributed pure GPU inference, large models may be split across multiple GPUs, i.e., tensor parallelism or pipelined parallelism. The additional parallelism and alternative branches introduced by speculative decoding significantly affect GPU memory usage and inter-GPU communication. If parameters are not chosen appropriately, uneven resource allocation may occur. Related techniques often do not fully consider the limitations of GPU memory and communication bandwidth, lacking end-to-end resource coordination mechanisms, resulting in unstable performance under different hardware configurations. Finally, there is a mismatch between draft length and validation resources. When the difficulty of requests or decoding branches within a batch varies significantly, a fixed draft length strategy can create a bottleneck effect. Some sequences may be validated quickly, while others require multiple rounds of validation due to excessively long drafts or low acceptance rates. As a result, the computational resources of the former are idle while waiting for the latter, leading to tail latency bottlenecks or lagging issues. Conversely, if the speculative step sizes of each sequence differ and there is a lack of coordination, uneven GPU load during the validation phase may occur. This mismatch between draft generation length and validation scheduling resources reduces overall throughput and causes significant tail latency fluctuations. Existing solutions often simply apply a uniform speculative step size to the entire batch, or even if adjustments are made for each sequence, there are no further restrictions, leading to excessive speculation in some sequences that slows down the overall process.

[0050] In summary, speculative decoding methods in related technologies lack the ability to adaptively adjust to system resources and workload states: parameter tuning mainly relies on offline experience or static configuration, and cannot be adjusted with dynamic changes in GPU memory utilization and computing / communication bandwidth usage; thus, under large-scale pure GPU distributed inference, problems such as insufficient resource utilization or bottleneck lag occur, affecting the stability and consistency of throughput improvement.

[0051] Therefore, there is an urgent need for a data processing method that can effectively achieve adaptive adjustment of system resources and workload status, thereby ensuring the stability and consistency of large model inference performance.

[0052] Please refer to Figure 1 This illustration shows a flowchart of a data processing method provided in one embodiment of this application. Specifically, the data processing method may include:

[0053] Step 101: Obtain hardware resource usage and draft history acceptance rate; wherein, the hardware resource usage is determined based on the operation of the GPU cluster system executing the inference of the large language model, and the draft history acceptance rate is determined based on the inference results of the large language model.

[0054] Step 102: Based on the hardware resource usage and the historical acceptance rate of the draft, use the inference acceleration strategy prediction model to obtain the speculative parameter adjustment strategy of the speculative decoding algorithm.

[0055] Step 103: Optimize the inference of the large language model based on the speculative parameter adjustment strategy.

[0056] It should be noted that a large language model can include a draft model and a validation model. The draft model can be a helper model. The validation model can be the original large language model, i.e., the target model. The validation model can be a model based on a tree-like attention mechanism.

[0057] It should be noted that hardware resource usage can include real-time GPU resource utilization. Hardware resource usage may include, but is not limited to, information on video memory usage, scheduling parallelism, communication bandwidth, communication latency, and computational load. The draft history acceptance rate can be the actual draft acceptance rate generated after inference processing by the large language model.

[0058] It should be noted that speculative parameters can include draft length, validation granularity, and tree width. Draft length can refer to the number of tokens L generated in parallel by the draft model in one pass. Validation granularity can refer to the step size of the validation model in processing the draft in one pass. Tree width can be the number of parallel candidate branches K per step, i.e., the number of parallel branches.

[0059] Understandably, when the large language model receives a new batch of inference requests, the speculative parameters for that batch of requests can first be initialized. The initial values ​​of these speculative parameters can be set based on historical statistics or simple rules. For example, high-confidence predictions on the first few drafts can trigger a larger initial draft length; otherwise, a conservative value is used. The current system load level, such as the number of concurrent requests and remaining GPU memory, is recorded for use in the initial decision-making of the inference acceleration strategy prediction model. If the main model and draft small models have not yet been loaded, they are loaded via the executor interface, and a KV cache paging index table data structure is established for the draft model and validation model instances.

[0060] It should be noted that in step 101, the initial speculative parameter information and profit information, or the speculative parameter information and profit information of the previous optimization processing round, can also be obtained.

[0061] In this way, by acquiring hardware resource usage and draft historical acceptance rates—where hardware resource usage is determined based on the operation of the GPU cluster system executing the large language model inference and draft historical acceptance rates are determined based on the inference results of the large language model—a speculative parameter adjustment strategy for the speculative decoding algorithm can be obtained using an inference acceleration strategy prediction model based on these hardware resource usage and draft historical acceptance rates. Based on this speculative parameter adjustment strategy, the inference of the large language model can be optimized. Since the speculative parameter adjustment strategy for the speculative decoding algorithm can be analyzed and determined based on the hardware resource usage and draft historical acceptance rates generated during the large language model inference process, the speculative parameters can be dynamically adjusted according to the real-time hardware operation, achieving adaptive adjustment of system resources and workload status, improving parameter adjustment flexibility, and thus ensuring the stability and consistency of the large model inference performance.

[0062] Optionally, in one possible implementation of this embodiment, the inference acceleration strategy prediction model may include a hardware-aware cost model and an acceptance rate prediction model. In step 102, firstly, hardware cost information can be obtained based on the hardware resource usage using the hardware-aware cost model. Secondly, draft revenue information can be obtained based on the historical draft acceptance rate using the acceptance rate prediction model. Thirdly, a speculative parameter adjustment strategy for the speculative decoding algorithm can be obtained based on the hardware cost information and the draft revenue information.

[0063] In this implementation, the inference acceleration strategy prediction model can be a model based on the extreme gradient boosting (XGBoost) algorithm.

[0064] In one specific implementation of this approach, the hardware resource usage is input into the hardware-aware cost model to perform cost estimation processing on the hardware resource usage, and output the hardware cost information corresponding to the hardware resource usage.

[0065] In this implementation, hardware cost information can characterize the system hardware loss cost of using the speculative decoding algorithm under the current conditions.

[0066] In another specific implementation of this method, the historical acceptance rate of the draft is input into the acceptance rate prediction model, the historical acceptance rate of the draft is predicted, and the draft revenue information corresponding to the historical acceptance rate of the draft is output.

[0067] In another specific implementation of this approach, firstly, a first benefit can be determined based on the hardware cost information and the draft benefit information. Secondly, the current second benefit can be obtained. Thirdly, based on preset benefit conditions, the first benefit, and the second benefit, a speculative parameter adjustment strategy for the speculative decoding algorithm can be obtained.

[0068] In this implementation, the first benefit can be the projected net benefit. The second benefit can include the current actual benefit and the projected net benefit from the previous period.

[0069] In one implementation scenario, the first benefit can be obtained based on the difference between the draft benefit information and the hardware cost information.

[0070] In this implementation, the preset benefit condition can be that the first benefit is greater than the second benefit.

[0071] In this implementation, the speculation parameter adjustment strategy may include adjusting speculation parameter information and maintaining the current speculation parameter information. Adjusting speculation parameter information may include increasing draft length and / or tree width, or decreasing draft length and / or tree width.

[0072] Another specific implementation is as follows: In response to the relationship between the first and second returns satisfying a preset return condition, the speculation parameter adjustment strategy of the speculation decoding algorithm is determined to be adjusting the speculation parameter information. In response to the relationship between the first and second returns not satisfying the preset return condition, the speculation parameter adjustment strategy of the speculation decoding algorithm is determined to be either maintaining the current speculation parameter information or adjusting the speculation parameter information.

[0073] Here, if the first payoff is greater than the second payoff, the draft length and / or tree width are increased based on the speculative parameter information output by the inference acceleration strategy prediction model. If the first payoff is less than or equal to the second payoff, the current speculative parameter information can be maintained. If the first payoff is less than or equal to the second payoff, the draft length and / or tree width can also be reduced based on the speculative parameter information output by the inference acceleration strategy prediction model.

[0074] In this implementation, the draft revenue information can be a value representing an estimated acceptance rate. The draft revenue information can be determined by obtaining the predicted rate of return based on the acceptance rate prediction model. The hardware cost information can be a value representing an estimated hardware expense.

[0075] In another specific implementation of this approach, in response to the hardware cost information being greater than a preset cost threshold, or the draft revenue information being less than a preset first revenue threshold, the draft length and / or tree width can be reduced to suspend speculative decoding processing. In response to the draft revenue information being greater than a preset second revenue threshold and the hardware cost information being less than a preset cost threshold, the draft length and / or tree width can be increased to maintain or enable speculative decoding processing, wherein the preset second revenue threshold is greater than the preset first revenue threshold.

[0076] Understandably, if all candidate drafts are acceptable in the current inference round and the GPU still has spare capacity, the draft length for the next round can be increased or the number of parallel branches can be increased, thus improving inference speed. If most draft tokens are rejected in the current round or GPU memory is insufficient, the draft length can be reduced or some parallel mechanisms can be temporarily disabled, i.e., tree attention can be turned off or the tree width K value can be reduced, prioritizing stable inference output. This decision can be made after the model validation is completed and before the next round of drafts begins, and the adjusted speculative parameter information can be distributed to achieve hot updates.

[0077] In another specific implementation of this approach, for the same batch of inference requests, in different processing rounds, when the draft length is equal to the preset length threshold, the draft length is not increased to prevent tailing.

[0078] In this way, the hardware resource usage can be processed through a hardware-aware cost model to obtain more accurate and effective hardware cost information. Furthermore, the acceptance rate prediction model can be used to obtain more accurate and effective draft revenue information based on the historical acceptance rate of the draft. Then, based on the hardware cost information and draft revenue information, a more accurate and effective speculative parameter adjustment strategy for the speculative decoding algorithm can be obtained. Through intelligent scheduling, the computational resources of both the draft model and the large model can be fully utilized, further optimizing the inference performance of the large language model.

[0079] Optionally, in one possible implementation of this embodiment, in step 102, the optimal solution of the inference acceleration strategy prediction model can be determined based on the hardware resource usage, draft history acceptance rate, objective function, and constraints. Then, based on the optimal solution of the inference acceleration strategy prediction model, the speculative parameter adjustment strategy of the speculative decoding algorithm can be obtained.

[0080] In this implementation, the objective function can be to maximize the net system benefit under the current hardware environment, as shown in formula (1):

[0081] (1)

[0082] Here, Gain(L,K) is the revenue term, i.e., the draft revenue information, predicted by the acceptance rate model, representing the expected time saved by successful speculation under parameters (L,K). Cost(L,K) is the cost term, i.e., the hardware cost information, predicted by the hardware-aware cost model based on XGBoost, representing the additional overhead incurred by executing this tree structure. L is the draft length, and K is the tree width, i.e., the number of parallel candidate branches.

[0083] here, , It is the probability that each node will be accepted, as output by the acceptance rate prediction model. The time saved per hit. , This refers to the usage of hardware resources, which can include hardware metrics such as GPU utilization, communication latency, and available video memory.

[0084] In this implementation, the constraints may include memory safety constraints, latency benefit positive constraints, and cross-sequence long-tail constraints.

[0085] Here, it is understandable that, in order to ensure system stability and response speed, the following constraints can be satisfied when searching for the optimal (L,K):

[0086] First, the memory safety constraint is shown in formula (2):

[0087] (2)

[0088] Here, executing the Eagle Tree Attention mechanism will consume additional GPU memory. This memory safety constraint can be used to ensure the peak GPU memory usage for prediction. The current safe value for video memory does not exceed the set safety threshold. For example, 90% of the total video memory should be allocated to prevent video memory overflow.

[0089] Secondly, the latency gain constraint is as shown in formula (3):

[0090] (3)

[0091] Here, when the predicted revenue item is significantly larger than the hardware cost item, The tolerance factor can be adjusted. This avoids meaningless and frequent parameter jittering under extreme hardware load.

[0092] Furthermore, the global batch constraint across sequences is shown in equation (4):

[0093] (4)

[0094] Here, in a batch inference request scenario... The draft length can be set. This is the step size for the slowest draft sequence. This allows us to set an upper limit based on the step size of the slowest sequence, preventing aggressive sequences from excessively consuming computing power and causing other sequences to lag behind.

[0095] In another specific implementation of this approach, the optimal solution of the prediction model can be obtained based on the inference acceleration strategy, and the output result can be obtained based on the output result, the current speculative parameters, and the current profit. The speculative parameter adjustment strategy of the speculative decoding algorithm can then be obtained.

[0096] Here, the current revenue can be the revenue information for the current draft. The current revenue can be determined based on the historical acceptance rate of the draft. The output can include the adjusted speculative parameters and the expected revenue. The expected revenue can be the new draft revenue information, and the adjusted speculative parameters can include the new draft length, tree width, etc.

[0097] For example, in the current speculative parameters, the draft length is 5, the tree width is 2 degrees, and the current payout is 8 seconds. If the output of the inference acceleration strategy prediction model is a draft length of 4, a tree width of 4, and an expected payout of 12 seconds, since the expected payout of 12 seconds is greater than the current payout of 8 seconds, then the draft length is adjusted to 4, and the tree width is adjusted to 4. If the output of the inference acceleration strategy prediction model is a draft length of 4, a tree width of 6, and an expected payout of 4 seconds, since the expected payout of 4 seconds is less than the current payout of 8 seconds, then the speculative parameter information does not need to be adjusted.

[0098] In this way, the optimal solution of the prediction model based on the inference acceleration strategy can be obtained to obtain the speculative parameter adjustment strategy of the speculative decoding algorithm, which further improves the reliability of speculative parameter adjustment.

[0099] It should be noted that the specific implementation process provided in this implementation method can be combined with various specific implementation processes provided in the foregoing implementation methods to implement the data processing method of this embodiment. Detailed descriptions can be found in the relevant content of the foregoing implementation methods, and will not be repeated here.

[0100] Optionally, in one possible implementation of this embodiment, in step 103, firstly, the strategy can be adjusted based on the speculative parameters to obtain speculative parameter information. Secondly, the inference of the large language model is optimized based on the speculative parameter information.

[0101] In one specific implementation of this approach, in response to the speculation parameter adjustment strategy of maintaining the current speculation parameter information, the current speculation parameter is obtained and used as the speculation parameter information.

[0102] In another specific implementation of this approach, in response to the speculation parameter adjustment strategy, the speculation parameter information is adjusted, and the adjusted speculation parameters are obtained so as to use the adjusted speculation parameters as the speculation parameter information.

[0103] In another specific implementation of this approach, in response to the speculative parameter adjustment strategy of maintaining the current speculative parameter information, the inference of the large language model is optimized based on the current speculative parameters.

[0104] Here, the current speculative parameter information can be the speculative parameters based on the current large language model inference processing. If the speculative parameter adjustment strategy is to maintain the current speculative parameter information, it can continue to be used as parameters for the speculative decoding algorithm to control the inference of the large language model.

[0105] In another specific implementation of this approach, in response to the speculative parameter adjustment strategy, the speculative parameter information is adjusted, the adjusted speculative parameters are obtained, and the inference of the large language model is optimized based on the adjusted speculative parameters.

[0106] In this way, by adjusting the strategy through speculative parameters, the reasoning speed of the large language model can be optimized without affecting the accuracy of the reasoning results, thereby further improving the reasoning performance of the large language model.

[0107] It should be noted that the specific implementation process provided in this implementation method can be combined with various specific implementation processes provided in the foregoing implementation methods to implement the data processing method of this embodiment. Detailed descriptions can be found in the relevant content of the foregoing implementation methods, and will not be repeated here.

[0108] Alternatively, in one possible implementation of this embodiment, Figure 2This is a schematic diagram illustrating an application scenario of the architecture of the data processing method provided in one embodiment of this application, such as... Figure 2 As shown, the application scenario includes a perception layer, a performance prediction layer, and an adaptive layer. The perception layer collects real-time data such as hardware resource usage and speculative decoding acceptance rate. Hardware resource usage includes hardware resource utilization (GPU load) and latency. The perception layer provides this real-time data to the performance prediction layer. The performance prediction layer, i.e., the inference acceleration strategy prediction model, can include an acceptance rate predictor and a hardware-aware cost model. The performance prediction layer can use the acceptance rate predictor and the hardware-aware cost model to estimate the expected benefits (acceptance rate improvement) and costs (hardware costs) of using speculative decoding under current conditions, obtaining prediction results and determining whether speculative decoding is worthwhile. If the acceptance rate is high and the cost is acceptable, speculative decoding is maintained or enabled (including using a tree-like attention mechanism to accelerate inference); conversely, if the acceptance rate is low or the cost is too high, speculative decoding is temporarily disabled. The adaptive layer can implement specific strategies based on the obtained adjustment strategy information, such as adjusting relevant parameters to enable / disable speculative decoding. The adaptive layer directly interacts with the underlying inference service and hardware resource management module, controlling the speculative decoding process according to policy instructions. For example, when the policy engine decides to pause speculative decoding, the controller sends a signal to disable the parallel inference thread of the draft model; when speculation needs to be enabled, the draft model is activated and executed according to the set configuration. Simultaneously, the operating parameters of the tree-like attention start / stop mechanism can be adjusted, such as the number of parallel branches and the attention threshold, to optimize the execution of speculative decoding in accordance with the current policy. The adaptive layer ensures that the decisions of the perception-prediction-adaptive architecture can quickly take effect in the actual system. The execution results of the adaptive layer can be fed back to the performance prediction layer, entering the next feedback loop.

[0109] Understandably, the draft model in the speculative decoding algorithm can generate candidate draft sequences for each inference request. A pipelined parallel deployment of small models may be employed, allowing draft generation to utilize multi-GPU computing power. Performance metrics during the draft stage are continuously acquired, including the inference time of the draft model, the load of the small model itself, and the acceptance rate of the currently generated draft sequence. An acceptance rate predictor evaluates the acceptance rate of the draft model's output at each step, providing real-time signals indicating whether to switch to verification early. The draft model stage also remains on standby to capture early stop instructions from the adaptive layer: if it is determined that the draft should stop, the executor interface can be immediately instructed to halt subsequent outputs of the draft model, locking the currently generated candidate draft sequences and preparing them for processing by the verification model. This ensures that the draft length L is not a fixed value but dynamically determined by the predictability of the actual content. During the adaptive layer's execution parameter update and verification model's verification phase, after the parameters of the candidate draft sequence or tree structure are updated, the verification model performs a forward propagation to score and verify the candidate drafts. If tree attention is enabled, the validation model processes tree-like batches in a single forward pass, enabling parallel validation of multiple sequences; otherwise, it validates draft sequences sequentially until a rejected draft is encountered. After validation, results such as which candidate drafts were accepted and at what point rejection occurred can be obtained. Based on these results, the acceptance rate predictor updates the historical acceptance rate of the sequence's drafts and provides the acceptance rate prediction for the next round. During the validation phase, the utilization of GPU hardware resources running the large model can also be obtained, such as how much VRAM was used in this round of validation and whether there were any waiting periods (communication or idle time) during computation, for use in the next round of data processing. The execution results of each round are fed back to form the input for the next round of perception. This loop continues iterating until the large model inference ends, i.e., after each round of dynamic adjustment until all request sequences are generated. In this way, a closed loop of perception-decision-execution can be formed throughout the entire large model inference process. Every few rounds, long-term metrics, such as recent average acceptance rate and throughput trends, can be referenced to calibrate the cost model and prevent error accumulation. For example, if a systematic deviation between the actual throughput and the model prediction is detected, the model parameters are adjusted to improve the prediction accuracy, making the parameter tuning decision more precise.

[0110] In a specific implementation of this approach, when constructing the inference acceleration strategy prediction model, the first step is to establish a performance prediction model for speculative decoding, i.e., the inference acceleration strategy prediction model, which comprehensively considers factors such as computational latency, memory overhead, and communication costs. This model can take the current system state and speculative parameter configuration as input and output an estimated cost / benefit metric. For example, based on the Goodput function or a similar utility function, the number of valid drafts validated per second can be defined as the optimization objective. The inference acceleration strategy prediction model needs to characterize: the time consumed by the draft model to generate L tokens, the time consumed by the large model to validate these tokens in parallel, and potential idle waiting or redundant computation losses. Through linear regression or more complex regression models, combined with offline profiling data and online monitoring data, the latency and number of valid drafts per inference step under different L and K configurations are estimated, thereby inferring the output per unit time (Goodput). This model can select parameter configurations based on the current system load and the predicted acceptance length. For example, in high-load batch scenarios, if the inference acceleration strategy prediction model predicts that further increasing the draft length will reduce Goodput, the strategy should appropriately reduce the draft length or even suspend speculation; conversely, in low-load scenarios, parallelism can be increased to improve throughput.

[0111] In another specific implementation of this approach, the inference acceleration strategy prediction model can include an acceptance rate predictor. Here, a lightweight online acceptance rate predictor can be pre-built. This predictor dynamically predicts the probability that a candidate draft will be accepted by the validation model at each draft stage. The confidence level of the draft model for the next candidate draft, i.e., the softmax probability or acceptance rate, can be used as a signal. When the confidence level drops to a certain threshold, it indicates that continuing generation is likely to result in incorrect predictions. The confidence level is evaluated in real-time as the draft model generates tokens one by one. If it falls below the threshold, the draft stage is terminated early and the large model is used for validation, thus dynamically determining the length of the current draft round. This acceptance rate-based feedback ensures that the draft model only generates more tokens when it is confident, avoiding blindly accumulating a fixed number and increasing the number of invalid candidate drafts. In addition to the acceptance rate threshold, more prediction signals can be incorporated, such as using the divergence between the previous layer outputs of the target large model and the output distribution of the draft model (e.g., KL divergence) to diagnose the stability of the current generation. When a high-uncertainty region is detected, the draft is shortened in a timely manner to reduce erroneous candidate drafts; conversely, when the sequence is stable, longer drafts are allowed to facilitate full parallelism. This acceptance rate predictor essentially provides closed-loop feedback for the speculative decoding process, adjusting subsequent strategies based on the acceptance results of each previous round, thus achieving "online adaptation".

[0112] In another specific implementation of this approach, a strategy can be adjusted based on speculative parameters to enable or disable the tree-based parallel verification mode as needed, adapting to different resource conditions and content requirements. Tree attention verifies multiple candidate branches in parallel at once, avoiding the latency accumulation caused by multiple rounds of sequential verification when the acceptance rate is low. However, tree-based verification also consumes more GPU memory and increases the computation of one forward pass. A tree attention start / stop strategy can be used: when GPU utilization is low during the verification phase, there is free batch processing space, or the probability of divergence in the current draft tokens is predicted to be high, tree-based verification can be enabled, allowing the large model to verify multiple possible branches at once, reducing the overhead of backtracking and recalculation; conversely, if system GPU memory is scarce or the acceptance rate of recent drafts is extremely high, tree attention can be disabled, and a single-path verification can be used to save resources. Tree attention should only be enabled when it is expected to reduce overall latency and resources allow. This on-demand usage leverages the advantages of tree parallelism while avoiding unnecessary overhead. When there are multiple possible output paths, tree-parallel one-time verification can improve effective throughput, while when the output height is fixed, there is no need to start complex tree structure verification.

[0113] Thus, through the aforementioned architecture, this solution achieves dynamic tuning of speculative decoding parameters in a distributed, pure GPU environment. It's equivalent to adding an adaptive controller layer on top of traditional large-model inference systems, adjusting the generation strategy in real-time based on system hardware conditions to maximize performance. This solution is compatible with various speculative decoding methods and model types—whether using a small draft head bound to the model like EAGLE3, an independent small model, or alternative techniques such as tree-based validation or Prompt caching—the policy engine can evaluate their applicability through a cost model and selectively enable them. Therefore, this dynamic tuning framework is versatile and can be seen as a higher-level integration and enhancement of existing inference acceleration methods, ensuring that regardless of the method used, intelligent switching based on load is possible in actual deployments, preventing performance degradation.

[0114] For example, the solution in this embodiment can be applied to inference scenarios of the Qwen3-235B large language model on a distributed cluster composed of 8×NVIDIA H100 GPUs. On the one hand, in a single-request (Batch=1) scenario, when a high GPU idle rate is detected, the draft length and parallel branches can be aggressively increased to fill the computing units. For example, for a long inference generation request, the initial round system may increase the draft length to, for example, 8 or 16, trying to make the large model verify as many tokens as possible each time. Since there is no cross-request interference in the single-request scenario, as long as the acceptance rate predictor judges the acceptance rate of the draft model output, such as the acceptance rate in the first few rounds reaching more than 90%, a large draft length can be maintained to achieve a speedup of nearly 2.5 times or more. In practical applications, for an inference output of length 512, the dynamic parameter adjustment strategy in the single-request scenario reduces latency by about 60% and increases throughput by about 150% compared to not performing speculative decoding.

[0115] On the other hand, in large-batch (Batch=32) scenarios, GPU utilization is already high, and the content of sequences within a batch may vary significantly. Initially, a medium draft length (e.g., L=4) is adopted for all draft sequences, and the acceptance rate is observed and fine-tuned for each sequence. If certain sequences consistently have a high acceptance rate over several rounds, the system will individually increase the L value for that sequence to 6 or 8; however, to prevent them from completing too quickly and causing other slower sequences to wait too long, a global step size cap is introduced, for example, not exceeding 8. For sequences with low acceptance rates, such as code generation requests where small models often have poor hit rates, the dynamic strategy will quickly reduce their draft length or even temporarily disable speculation, allowing large models to take over for a period of time to avoid the draft models continuously wasting computing power. This on-demand adjustment ensures that even in large-batch environments with mixed tasks, each sequence can utilize its own resources, preventing individual tasks from severely slowing down the overall process due to the use of uniform static parameters. Actual testing shows that, with Batch=32 and requests consisting of a mix of dialogue and code, the dynamic approach reduces tail latency by about 30% compared to the fixed-parameter approach, while increasing throughput by about 20%. Timely degradation handling of low-acceptance-rate tasks avoids global blocking.

[0116] Thus, by adopting the solution in this embodiment, the system status can be perceived in real time and the overhead and benefits of different parameter configurations for speculative decoding can be predicted by introducing an online prediction model and feedback mechanism, thereby dynamically adjusting the parameters to achieve optimal resource allocation.

[0117] Furthermore, the solution presented in this embodiment can map key parameters of speculative decoding, such as draft length L, number of parallel candidates K, and verification step size, to the real-time resource utilization of the GPU through a hardware-aware cost model, enabling online adjustment of speculative decoding parameters. Compared to existing static or simple heuristic-based solutions, this approach can automatically adjust the speculative intensity based on current GPU memory availability and computation / communication load, maximizing hardware utilization efficiency while ensuring accuracy, and can be widely applied to LLM inference acceleration at different scales.

[0118] Furthermore, by employing the scheme in this embodiment, an online acceptance rate predictor can be used to monitor indicators such as the confidence level or KL divergence of each draft prediction in the draft model, thereby predicting the acceptance probability of the draft in real time. In conjunction with feedback closed-loop control, this feedback can be used to dynamically decide the draft generation length, thus enabling real-time fine-tuning of parameters during the speculation process.

[0119] Furthermore, the solution described in this embodiment can fully utilize the computing resources of both small and large models through intelligent scheduling, reducing memory fragmentation or prolonged GPU idle time. Simultaneously, with speculative acceleration enabled, dynamic adjustments only affect computational scheduling and do not sacrifice the accuracy of the large model's generated results or alter the final output content verified by the large model. This ensures complete consistency with the large model's output text compared to when the speculative decoding algorithm is not used, effectively preventing a decline in the inference performance of the large model.

[0120] It should be noted that the specific implementation process provided in this implementation method can be combined with various specific implementation processes provided in the foregoing implementation methods to implement the data processing method of this embodiment. Detailed descriptions can be found in the relevant content of the foregoing implementation methods, and will not be repeated here.

[0121] Figure 3 A structural block diagram of a data processing apparatus according to an embodiment of this application is shown, as follows: Figure 3 As shown. The data processing apparatus 300 of this embodiment may include an acquisition unit 301, a prediction unit 302, and an optimization unit 303. The acquisition unit 301 is used to acquire hardware resource usage and draft history acceptance rate; wherein the hardware resource usage is determined based on the operating status of the GPU cluster system executing the inference of the large language model, and the draft history acceptance rate is determined based on the inference results of the large language model; the prediction unit 302 is used to obtain a speculative parameter adjustment strategy for the speculative decoding algorithm based on the hardware resource usage and draft history acceptance rate, using an inference acceleration strategy prediction model; the optimization unit 303 is used to optimize the inference of the large language model based on the speculative parameter adjustment strategy.

[0122] Optionally, in one possible implementation of this embodiment, the inference acceleration strategy prediction model includes a hardware-aware cost model and an acceptance rate prediction model. The prediction unit 302 is used to obtain hardware cost information based on the hardware resource usage using the hardware-aware cost model; obtain draft revenue information based on the historical draft acceptance rate using the acceptance rate prediction model; and obtain a speculative parameter adjustment strategy for the speculative decoding algorithm based on the hardware cost information and the draft revenue information.

[0123] Optionally, in one possible implementation of this embodiment, the prediction unit 302 is used to determine a first benefit based on the hardware cost information and the draft benefit information; obtain the current second benefit; and obtain a speculative parameter adjustment strategy for the speculative decoding algorithm based on preset benefit conditions, the first benefit, and the second benefit.

[0124] Optionally, in one possible implementation of this embodiment, the prediction unit 302 is used to determine the optimal solution of the inference acceleration strategy prediction model based on the hardware resource usage, the historical acceptance rate of the draft, the objective function, and the constraints; and to obtain the speculative parameter adjustment strategy of the speculative decoding algorithm based on the optimal solution of the inference acceleration strategy prediction model.

[0125] Optionally, in one possible implementation of this embodiment, the optimization unit 303 is used to adjust the strategy based on the speculative parameters to obtain speculative parameters; and to optimize the inference of the large language model based on the speculative parameter information.

[0126] Optionally, in one possible implementation of this embodiment, the optimization unit 303 is configured to, in response to the speculation parameter adjustment strategy of maintaining the current speculation parameter information, obtain the current speculation parameter and use the current speculation parameter as the speculation parameter information; and in response to the speculation parameter adjustment strategy of adjusting the speculation parameter information, obtain the adjusted speculation parameter and use the adjusted speculation parameter as the speculation parameter information.

[0127] In this embodiment, the environmental awareness information of the social application can be acquired by the acquisition unit. The environmental awareness information includes text information, time sequence information, and metadata information. The identification unit uses a preset mutation identification engine to identify and process the environmental awareness information. The determination unit determines whether a mutation event exists based on the result of the identification and processing. In response to the existence of the mutation event, the fine-tuning unit performs fine-tuning training on the pre-trained policy model to obtain a fine-tuned policy model. The response unit determines the target policy information for responding to the mutation event based on the fine-tuned policy model. Since the speculative parameter adjustment strategy of the speculative decoding algorithm can be analyzed and determined based on the hardware resource usage and the historical acceptance rate of the draft generated by the inference during the large language model inference process, the speculative parameters can be dynamically adjusted according to the real-time hardware operation, realizing adaptive adjustment of system resources and workload status, improving the flexibility of parameter adjustment, and thus ensuring the stability and consistency of the large model inference performance.

[0128] The technical solution of this application involves the collection, storage, use, processing, transmission, provision, and disclosure of user personal information, such as user image and attribute data, which comply with relevant laws and regulations and do not violate public order and good morals.

[0129] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0130] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0131] like Figure 4As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0132] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of displays, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0133] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the data processing methods described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0134] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0139] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0140] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0141] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: The hardware resource usage and draft history acceptance rate are obtained; wherein the hardware resource usage is determined based on the operation of the GPU cluster system that performs inference for the large language model, and the draft history acceptance rate is determined based on the inference results of the large language model. Based on the hardware resource usage and the historical acceptance rate of the draft, the speculative parameter adjustment strategy of the speculative decoding algorithm is obtained by using the inference acceleration strategy prediction model. Based on the aforementioned speculative parameter adjustment strategy, the inference process of the large language model is optimized, wherein... The method for adjusting the speculative parameters of the speculative decoding algorithm, based on the hardware resource usage and historical draft acceptance rate, and using an inference acceleration strategy prediction model, includes: Based on the hardware resource usage, draft historical acceptance rate, objective function, and constraints, the optimal solution of the inference acceleration strategy prediction model is determined; the constraints include memory security constraints, latency benefit positive constraints, and cross-sequence long-tail constraints. Based on the optimal solution of the inference acceleration strategy prediction model, the output results are obtained, which include the adjusted speculative parameters and expected returns. Based on the output results, current speculative parameters, and current returns, the speculative parameter adjustment strategy of the speculative decoding algorithm is obtained.

2. The method according to claim 1, characterized in that, The inference acceleration strategy prediction model includes a hardware-aware cost model and an acceptance rate prediction model. Based on the hardware resource usage and the historical acceptance rate of the draft, the inference acceleration strategy prediction model is used to obtain the speculative parameter adjustment strategy for the speculative decoding algorithm, including: Based on the hardware resource usage, hardware cost information is obtained using the hardware-aware cost model. Based on the historical acceptance rate of the draft, the acceptance rate prediction model is used to obtain draft revenue information; Based on the hardware cost information and draft profit information, a speculative parameter adjustment strategy for the speculative decoding algorithm is obtained.

3. The method according to claim 2, characterized in that, The method for adjusting the speculative parameters of the speculative decoding algorithm based on the hardware cost information and draft profit information includes: Based on the hardware cost information and the draft revenue information, the first revenue is determined; Obtain the current second benefit; Based on preset profit conditions, the first profit and the second profit, a speculative parameter adjustment strategy for the speculative decoding algorithm is obtained.

4. The method according to claim 1, characterized in that, The optimization of the inference of the large language model based on the speculative parameter adjustment strategy includes: Based on the aforementioned speculative parameter adjustment strategy, speculative parameter information is obtained; Based on the speculative parameter information, the inference of the large language model is optimized.

5. The method according to claim 4, characterized in that, The process of obtaining speculative parameter information based on the speculative parameter adjustment strategy includes: In response to the speculation parameter adjustment strategy of maintaining the current speculation parameter information, the current speculation parameter is obtained and used as the speculation parameter information; In response to the speculation parameter adjustment strategy, which is to adjust the speculation parameter information, the adjusted speculation parameters are obtained and used as the speculation parameter information.

6. A data processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire hardware resource usage and draft history acceptance rate; wherein, the hardware resource usage is determined based on the operation of the GPU cluster system that performs inference of the large language model, and the draft history acceptance rate is determined based on the inference results of the large language model; The prediction unit is used to obtain the speculative parameter adjustment strategy of the speculative decoding algorithm based on the hardware resource usage and the historical acceptance rate of the draft, using the inference acceleration strategy prediction model. An optimization unit is used to optimize the inference of the large language model based on the strategy of adjusting the speculative parameters. The prediction unit is used to determine the optimal solution of the inference acceleration strategy prediction model based on the hardware resource usage, historical draft acceptance rate, objective function, and constraints. The constraints include memory safety constraints, latency benefit positive constraints, and cross-sequence long-tail constraints. Based on the optimal solution of the inference acceleration strategy prediction model, an output result is obtained, which includes adjusted speculative parameters and expected returns. Based on the output result, current speculative parameters, and current returns, a speculative parameter adjustment strategy for the speculative decoding algorithm is obtained.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images