Inference optimization method of pre-trained model and electronic device

Abstract

The application discloses a pre-training model inference optimization method and an electronic device, relates to the inference technology field of a pre-training model, and determines the priorities of each subtask based on the load data of multiple task stages of the pre-training model, and then allocates a calculation unit to the subtask according to the priority of the subtask, dynamically decouples each task stage, realizes efficient allocation of calculation resources, and improves resource utilization. On the other hand, the application transmits the first data of two adjacent task stages through a double buffering mechanism, which is beneficial to alleviating the low interlayer data transmission efficiency problem in the related art and improving the hardware bandwidth utilization rate. Therefore, the technical problems of low idle resource utilization rate of memory bandwidth and low interlayer transmission efficiency are solved, and the technical effects of improving resource utilization rate and bandwidth utilization rate are achieved.

Description

Technical Field

[0001] This application relates to the field of inference technology for pre-trained models, and more particularly to an inference optimization method and electronic device for pre-trained models. Background Technology

[0002] With the rise and development of artificial intelligence technology, its application areas are becoming increasingly widespread. Different business scenarios have different requirements for prediction efficiency, which continuously drives the optimization of the inference efficiency of pre-trained models. The inference process of a pre-trained model can usually be divided into multiple task stages, which are executed sequentially.

[0003] In related technologies, there are various inference optimization schemes for pre-trained models. Some schemes adopt synchronous pipeline designs, binding each task stage into a fixed process. This scheme suffers from low resource utilization due to idle memory bandwidth during the execution of some task stages. Other schemes use asynchronous pipelines to split the inference process of the pre-trained model into tasks, with multiple task stages executed in parallel and resources allocated through batch processing. This scheme suffers from uneven resource allocation due to single-task resource squeezing and low efficiency in inter-layer transmission and communication. Summary of the Invention

[0004] This application provides a method and system for optimizing the inference of a pre-trained model, so as to at least solve the problem of low resource utilization in the inference process of pre-trained models in related technologies.

[0005] This application provides an inference optimization method for a pre-trained model, comprising: inputting data to be predicted into a pre-trained model to obtain load data for multiple task stages of the pre-trained model; wherein the load data includes the current task stage, available resources, and the time priority of the task stage; the task stage includes a first subtask and a second subtask; determining a first priority of the first subtask and a second priority of the second subtask based on the load data; allocating the first subtask to a first computing unit based on the first priority to execute the first subtask and obtain first data; determining a second computing unit to be allocated to the second subtask based on the second priority; transmitting the first data to the target layer of the second computing unit through a double buffering mechanism to execute the second subtask based on the first data at the target layer to obtain the inference result of the pre-trained model. This application determines the priority of each subtask based on the load data of multiple task stages of the pre-trained model, and then allocates computing units to the subtasks according to their priorities. By dynamically decoupling each task stage, it achieves efficient allocation of computing resources and improves resource utilization. On the other hand, this application transmits the first data of two adjacent task stages through a double buffering mechanism, which helps to alleviate the problem of low data transmission efficiency between layers in related technologies and improves hardware bandwidth utilization.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the inference optimization method of any of the above-described pre-trained models.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the inference optimization method of any of the above-described pre-trained models.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the inference optimization method for any of the pre-trained models described above.

[0009] This application determines the priority of each subtask based on the load data of multiple task stages from a pre-trained model. Then, computational units are allocated to the subtasks according to their priorities. By dynamically decoupling the task stages, efficient allocation of computational resources is achieved, improving resource utilization. Furthermore, this application uses a double-buffering mechanism to transmit the first data from two adjacent task stages, which helps alleviate the problem of low inter-layer data transmission efficiency in related technologies and improves hardware bandwidth utilization. Therefore, it solves the technical problems of low utilization of idle memory bandwidth and low inter-layer transmission efficiency, achieving the technical effect of improving resource and bandwidth utilization. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating an inference optimization method for a pre-trained model provided in an embodiment of this application;

[0012] Figure 2 This is a schematic diagram of the architecture of a dynamic adaptive asynchronous pipeline system provided in an embodiment of this application;

[0013] Figure 3 This is a schematic diagram of the structure of a dynamic adaptive asynchronous pipeline system provided in an embodiment of this application;

[0014] Figure 4 A flowchart illustrating the priority scheduling process provided in an embodiment of this application;

[0015] Figure 5A flowchart illustrating the asynchronous tensor transfer process provided in an embodiment of this application;

[0016] Figure 6 A flowchart illustrating the speculative execution process provided in this application embodiment;

[0017] Figure 7 A flowchart illustrating the credit score calculation and resource adjustment process provided in this application embodiment;

[0018] Figure 8 A flowchart illustrating the system feedback process provided in an embodiment of this application;

[0019] Figure 9 This is a schematic diagram of the distributed multi-node deployment architecture provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0021] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] First, let me explain the terms used in this application:

[0024] The Dynamic Adaptive Asynchronous Pipeline (DAAP), proposed in this application, is the core architecture for optimizing pre-trained model inference. It solves the problems of low resource utilization and large latency fluctuations in traditional synchronous pipelines by dynamically decoupling the pre-filling and decoding stages and by multi-module collaborative adaptive adjustment.

[0025] The Priority-Based Adaptive Scheduler (PAS), the control center of the DAAP architecture, is responsible for real-time sensing of system load and dynamic allocation of computing resources. It is the core module for achieving adaptive adjustment of task priorities.

[0026] Asynchronous Tensor Passports (ATP) is a data transmission channel in the DAAP architecture. It decouples computation and communication through an innovative metadata passport combined with a double buffering mechanism, solving the problems of data corruption and buffer overflow in traditional asynchronous transmission.

[0027] The Speculative Execution Buffer (SEB) is an intelligent prediction module in the decoding phase of the DAAP architecture. It generates candidate output sequences in advance through a lightweight auxiliary model, reducing actual decoding time and optimizing LLM inference latency.

[0028] The Dynamic Credit System (DCS) is a resource regulator in the DAAP architecture. It achieves dynamic resource balance across computing units by quantifying device resource pressure, calculating credit values, and globally synchronizing adjustments, thus avoiding system overload or resource idleness.

[0029] Key-Value Cache (KV Cache);

[0030] Earliest Deadline First (EDF) takes precedence.

[0031] Pre-training is a strategy for training deep learning models. Its core lies in using large-scale datasets to initially train the model, enabling it to learn general feature representations. This process is similar to the basic learning stage humans undergo before learning new knowledge, accumulating experience through extensive reading and observation.

[0032] Pre-trained language models generally refer to language model training tasks designed based on large-scale corpora (including language training materials such as sentences and paragraphs). A large-scale neural network algorithm structure is trained to learn and implement the model, resulting in a pre-trained language model with its parameters. Subsequent tasks can then use this model for feature extraction or task fine-tuning to achieve specific objectives. The idea behind pre-training is to first train a set of model parameters for one task, then use these parameters to initialize the network model parameters, and finally use the initialized network model to train other tasks, obtaining models adapted for those tasks. By pre-training on large-scale corpora, neural language representation models can learn powerful language representation capabilities, extracting rich syntactic and semantic information from text. Pre-trained language models can provide tokens containing rich semantic information and sentence-level features for downstream tasks. Fine-tuning can also be performed directly on the pre-trained model for downstream tasks, conveniently and quickly obtaining downstream-specific models.

[0033] The neural network algorithm structure used to train the pre-trained language model can be CNN, RNN, LSTM, etc., or it can be a model built with attention networks, such as Transformer, BERT, GPT, Clip, etc. This application does not limit it. An attention network is a network model that uses an attention mechanism for training. This model extracts more important feature information from the input sequence by assigning different weights to each part of the input sequence, so that the model can finally obtain a more accurate output.

[0034] The inference process of a pre-trained model is typically divided into two stages: pre-filling and decoding. The pre-filling stage computes a key-value cache (KV cache) of the input sequence, while the decoding stage generates the output sequence autoregressively based on this cache. Traditional inference systems employ a synchronous pipeline design, leading to mutual blocking between the two stages and low utilization of computational resources. Furthermore, existing methods lack dynamic resource allocation mechanisms when handling mixed workloads (such as simultaneous new requests and continuous decoding requests), resulting in idle GPU cores or memory bandwidth bottlenecks.

[0035] In the field of parallel computing, layer-wise parallelism has been proven to effectively decompose the computational load of a model, but its application in inference scenarios is still limited by static task partitioning and synchronous communication overhead. Recent research has attempted to improve throughput through asynchronous pipelines (such as the Seesaw and TAG architectures), but these solutions have failed to solve the dynamic load balancing problem between the pre-filling and decoding stages.

[0036] Traditional pre-trained model inference systems employ a synchronous pipeline design, binding the pre-filling and decoding stages into a fixed process:

[0037] (1) Pre-filling stage: The KV cache of the input sequence is computed centrally, and all computing resources wait synchronously for this stage to complete;

[0038] (2) Decoding stage: Output is generated token by token based on KV Cache. Each generation step requires waiting for the previous step to complete, and new requests must queue up and wait for the current decoding task to finish before entering the pre-filling stage.

[0039] (3) Parallel strategy: Static inter-layer parallelism is adopted, and the model layers are fixedly allocated to different GPUs. Data transmission between layers needs to wait synchronously (such as using the All Reduce protocol).

[0040] Asynchronous pipeline architectures, such as Seesaw and TAG, have the following drawbacks:

[0041] (1) Low resource utilization: The characteristics of the pre-filling (computation-intensive) and decoding (memory-intensive) stages are mismatched, resulting in idle GPU cores or memory bandwidth (e.g., the GPU computing unit utilization is only 30%-50% during decoding).

[0042] (2) Large latency fluctuations: Long sequence decoding tasks will block the pre-filling of new requests, resulting in increased latency for short requests (e.g., a 100 token request may be blocked for more than 1 second by a 1000 token decoding task).

[0043] (3) Strong cross-stage dependency: Synchronous communication protocols cause inter-layer data transmission to wait for all devices to be ready, and communication overhead accounts for 25%-40% of the total delay.

[0044] Recent research employs an asynchronous pipeline to split the pre-filling and decoding stages, allowing the two stages to execute in parallel:

[0045] (1) The pre-filling task and the decoding task occupy independent computing resources, and priority is divided by static batch processing (such as PipeDec using a fixed batch size).

[0046] (3) Inter-layer communication uses simplified asynchronous transmission, but it is not optimized for the dynamic characteristics of KV Cache (such as fixed buffer size).

[0047] (4) Speculative execution mechanisms (such as Asteroid) use a fixed threshold to verify candidate sequences and do not consider real-time load changes.

[0048] The asynchronous pipeline splitting scheme has the following disadvantages:

[0049] (1) Rigid resource allocation: Static batch processing may cause high-priority new requests to be squeezed out by low-priority long decoding tasks;

[0050] (2) Limited communication efficiency: lack of fine-grained metadata management, asynchronous transmission is prone to data corruption or buffer overflow;

[0051] (3) Poor predictive adaptability: Fixed verification threshold is prone to incorrect prediction under high load (increasing retry cost), and under low load, resources are not fully utilized (missing acceleration opportunities).

[0052] Based on this, this invention proposes an innovative Dynamic Adaptive Asynchronous Pipeline (DAAP) architecture specifically designed to address the efficiency bottleneck in the inference process of pre-trained models. The core value of this architecture lies in its intelligent allocation and efficient utilization of computing resources through dynamic decoupling of the pre-filling and decoding stages. In existing technologies, the strong coupling between the pre-filling and decoding stages leads to low resource utilization and significant latency fluctuations, while DAAP effectively solves these problems through four key technological innovations.

[0053] To address the shortcomings of existing technologies, this invention solves the following problems: unbalanced dynamic load during the pre-filling and decoding stages, resulting in low utilization of computing resources; high synchronization overhead for inter-layer data transmission, leading to insufficient hardware bandwidth utilization; significant latency fluctuations during the decoding stage, especially in mixed load scenarios; and the lack of a global resource coordination mechanism, making it unable to adapt to changes in memory and computing pressure.

[0054] On the one hand, embodiments of this application provide an inference optimization method for a pre-trained model, and the method is described in detail in conjunction with the execution flow of the inference optimization method for a pre-trained model.

[0055] Reference Figure 1 The inference optimization method for the pre-trained model provided in this application includes the following steps:

[0056] Step S100: Input the data to be predicted into the pre-trained model and obtain the load data of multiple task stages of the pre-trained model; wherein, the load data includes the current task stage, available resources and the time priority of the task stage; the task stage includes a first subtask and a second subtask.

[0057] Step S200: Determine the first priority of the first subtask and the second priority of the second subtask based on the load data;

[0058] Step S300: According to the first priority, the first subtask is assigned to the first computing unit so that the first subtask is executed by the first computing unit to obtain the first data;

[0059] Step S400: Based on the second priority, determine the second computing unit to be assigned to the second subtask; transmit the first data to the target layer of the second computing unit through a double buffering mechanism, so that the second subtask is executed on the target layer based on the first data to obtain the inference result of the pre-trained model.

[0060] In this application, the pre-trained model can be any artificial intelligence model; the data to be predicted can be text data, which is used for data inference through the pre-trained model; the data to be predicted can also be video data, which is used for preference inference through the pre-trained model; the data to be predicted can be data of any form, and this application does not specifically limit the format of the data to be predicted or the purpose and type of the pre-trained model. This application breaks down the inference process of the model to be predicted into multiple task stages to execute the inference process, and collects load data for multiple task stages. The priority of the request corresponding to each task stage is calculated through the load data, and each task stage is allocated to the computing unit based on the calculated priority. That is, the first priority of this application is the request priority of the first sub-task, and the second priority is the request priority of the second sub-task. The load data in this application includes the current task stage, available resources, and the time priority of the task stage. Among them, the current task stage is the task stage in which the current request is located, which can be, for example, a pre-filling stage or a decoding stage. Available resources refer to the resource status of the original computing unit, which is the computing unit occupied by each task stage before resource scheduling, i.e., the current computing unit. The time priority is the execution priority of the request corresponding to the task stage, which to a certain extent represents the urgency of the request.

[0061] In some embodiments, the task phase of this application includes a first subtask and a second subtask. For example, the first subtask may be a task in the pre-filling phase, and the second subtask may be a task in the decoding phase. This application determines a first priority for the first subtask based on its load data; and determines a second priority for the second subtask based on its load data. This application assigns the first subtask to a first computing unit based on its first priority; and assigns the second subtask to a second computing unit based on its second priority. It should be noted that the first and second computing units in this application can be different computing units or the same computing unit. The first and second computing units are used for descriptive convenience to distinguish the execution processes of the first and second subtasks and are not intended to limit the specific numbering of the computing units. The computing unit in this application can be a GPU, a GPU group composed of multiple GPUs, or other types of units that perform computation, transmission, and storage.

[0062] Specifically, regarding the scheduling process of the two subtasks based on first and second priorities, the two subtasks (pre-filling and decoding stages) are normally assigned to the same computing unit (such as the same GPU), but can be split into different computing units in specific scenarios. Priority calculation does indeed set differentiated strategies for the characteristics of the two stages, and whether they are assigned to the same unit depends on a trade-off between data dependencies and resource efficiency. Understandably, pre-filling and decoding are consecutive stages of the pre-trained model with strong data dependencies: the core of the pre-filling stage is the output "KV Cache" (attention key-value cache of the input sequence), and the decoding stage reuses this KV Cache for each token generation step. If the two stages are assigned to different computing units, the KV Cache needs to be transferred via PCIe or network (e.g., from GPU1 to GPU2), and LLM KV Caches are typically GB in size (e.g., a 70B model processing 1k tokens requires approximately 10GB of KV Cache). Cross-unit transfer introduces hundreds of milliseconds of latency, offsetting the benefits of parallel acceleration.

[0063] Therefore, the default strategy of the PAS module (task scheduling) is to bind the pre-filling and decoding stages of the same task to the same computing unit. By prioritizing requests, it ensures the continuity of stages within the same unit. In this case, the priority focuses on the overall urgency of the task (e.g., the real-time nature of user dialogues, which can be achieved through time priority), rather than individually splitting stage priorities. Of course, under high load, it is permissible to split the stage into different units. Specifically, when the computing unit load is too high (e.g., the computing unit utilization is close to the threshold of 80%), stage splitting scheduling is triggered, and the decoding stage is scheduled to other idle units.

[0064] Therefore, the method provided in this application also includes:

[0065] If the first utilization rate of the original computing unit is greater than or equal to the preset resource threshold, the second computing unit is not the same computing unit as the first computing unit; wherein, the preset resource threshold is the upper limit of utilization to balance data transmission and waiting.

[0066] If the first utilization rate of the original computing unit is less than the preset resource threshold, the second computing unit and the first computing unit are the same computing unit.

[0067] Reference Figure 2 As shown, the DAAP system provided in this application consists of four main modules that work together to achieve efficient LLM inference:

[0068] (1) Priority Adaptive Scheduler (PAS): As the control center of the system, it is responsible for monitoring the load status of each pipeline stage in real time and dynamically allocating computing resources according to preset strategies. This module maintains real-time communication with all other modules to ensure the overall coordinated operation of the system.

[0069] (2) Asynchronous Tensor Passport (ATP) Communication System: This system serves as a data transmission channel connecting various computing units. It employs an innovative metadata management mechanism to decouple computation from communication. The system is deeply integrated with underlying hardware (such as NVLink or InfinityFabric) to maximize the utilization of hardware bandwidth.

[0070] (3) Speculative Execution Buffer (SEB): An intelligent prediction module located in the decoding stage, which generates possible output sequences in advance through a lightweight auxiliary model, significantly reducing the actual decoding time. This module is updated synchronously with the main model to ensure the accuracy of the prediction.

[0071] (4) Dynamic Credit System (DCS): The system's resource regulator, which achieves global resource balance by quantifying the resource pressure of each computing unit. This module works closely with the scheduler PAS and the communication system ATP to prevent system overload or resource idleness. Of course, in this application, ATP undertakes the core function of "cross-computing unit data communication".

[0072] It is understandable that if the first computing unit and the second computing unit are the same computing unit, the first data will be transmitted to the target layer; if the first computing unit and the second computing unit are not the same computing unit, the first data will be transmitted to the target layer of the second computing unit.

[0073] This application executes a first subtask through a first computing unit to obtain first data, and then transmits the first data to the target layer of the second computing unit through a double-buffering mechanism. The target layer is the model layer corresponding to the second subtask. By setting up two independent buffers, this application allows the data loading, data transmission, and computation to be executed in parallel, which helps alleviate the blocking problem caused by single-buffered computation waiting for transmission in related technologies. Of course, this application does not limit the specific number of buffers; those skilled in the art can set up multiple buffers to load and transmit data simultaneously as needed, improving inter-layer transmission efficiency. This application improves inter-layer transmission efficiency and bandwidth utilization through a double-buffering mechanism. Furthermore, this application uses multi-parameter load data to calculate priorities and perform real-time precise scheduling of computing resources, improving resource utilization.

[0074] Optionally, the second subtask is performed based on the first data to obtain the inference result of the pre-trained model, including:

[0075] The first data is input into the prediction model to predict the decoded output, thus obtaining the candidate sequence;

[0076] If the matching probability between the candidate sequence and the first data is greater than or equal to a preset probability threshold, the candidate sequence is taken as the inference result;

[0077] Alternatively, if the matching probability between the candidate sequence and the first data is less than a preset probability threshold, a decoding task is performed based on the first data to obtain the inference result; wherein, the preset probability threshold is the minimum acceptable prediction accuracy of the prediction model.

[0078] In some embodiments, this application predicts the execution output of the second subtask using a prediction model to obtain a candidate sequence; and calculates the relationship between the matching probability of the candidate sequence and the input first data and a preset probability threshold to determine whether to use the candidate sequence as the final inference result. It is understood that the prediction model in this application can be any artificial intelligence model that implements the inference of the second subtask. For example, the second subtask is decoding, and decoding inference is performed using the prediction model. The preset probability threshold is a preset value used to characterize the lower limit of the acceptable prediction accuracy of the prediction model. In some embodiments, those skilled in the art can predetermine the specific value of the prediction probability threshold according to actual needs. Specifically, conventional decoding processes consume a large amount of memory. In this application, the prediction model can choose a lightweight transformer model to reduce the memory consumption of the decoding stage, reduce computational overhead, and improve resource utilization.

[0079] Optionally, after allocating the first subtask to the first computing unit according to the first priority, the method further includes:

[0080] Obtain the cumulative duration; wherein, the cumulative duration is the time elapsed since the last resource scheduling;

[0081] If the cumulative duration is greater than or equal to the scheduling period, the cumulative duration is cleared to zero, and the load data of multiple task stages of the pre-trained model is returned to determine the priority based on the load data so as to reschedule the computing resources; wherein, the scheduling period is the preset duration between two consecutive resource scheduling.

[0082] This application designs the scheduling process as periodic scheduling, adjusting the interval between two consecutive scheduling operations by regulating the scheduling period. The scheduling period can be set according to actual needs; for example, it can be adjusted based on the current system load. The system of this application includes at least one of a first computing unit and a second computing unit. By periodically updating the computing priority based on load data and scheduling computing units based on the priority, this application improves the accuracy of scheduling, thereby improving the accuracy of resource allocation.

[0083] Optionally, before clearing the accumulated duration to zero if the accumulated duration is greater than or equal to the scheduling period, the method further includes:

[0084] The base duration is determined based on the earliest deadline priority algorithm;

[0085] Obtain the average load of all computing units, and determine a first coefficient based on the first load and the average load; wherein the first coefficient is negatively correlated with the average load, and the first load is the maximum allowable load of the system;

[0086] The scheduling period is determined based on the product of the first coefficient and the base duration.

[0087] This application updates the scheduling cycle based on the load status of the system (all computing units). First, a base duration is determined using the earliest deadline first algorithm. Alternatively, those skilled in the art can use other feasible algorithms such as the minimum relaxation time first algorithm to determine the base duration, which serves as the baseline scheduling cycle. The average load of the system is collected, and a first coefficient is determined based on the first load and the average load. The first coefficient is negatively correlated with the average load, meaning it decreases as the average load increases; conversely, it is positively correlated with the first load. Those skilled in the art can characterize the relationship between the first coefficient and the changes in the first load and average load using a preset formula; they can also characterize this relationship using a preset table; or they can characterize this relationship using a preset curve. The first load is the upper limit of the system's allowed load. Finally, the scheduling cycle is updated based on the product of the first coefficient and the base duration, thereby balancing load occupancy.

[0088] Specifically, the scheduler employs a modified EDF (Earliest Deadline First) algorithm, but incorporates a dynamic time slice mechanism. The duration of each scheduling period T is determined by the following formula:

[0089] (1-1)

[0090] in, Base time slice / base duration (default 1ms); The maximum allowable load / first load of the system; The current average load, The first coefficient; , To improve system utilization, This refers to the amount of free memory. This represents the total memory usage. This dynamic time-slice mechanism can reduce scheduling overhead under high load and increase task continuity under low load. By adjusting the scheduling cycle based on load conditions, it helps to balance the accuracy of resource scheduling with the resource load.

[0091] Optionally, before clearing the accumulated duration to zero if the accumulated duration is greater than or equal to the scheduling period, the method further includes:

[0092] If the system load is the first load, the scheduling period is determined to be the first duration;

[0093] Alternatively, if the system load is the second load, the scheduling period is determined to be the second duration; wherein the first load is higher than the second load, and the first duration is less than the second duration.

[0094] This application can also adjust the scheduling cycle directly through scheduling strategies based on the system load. Specifically, under high load, the scheduling cycle is reduced, and resource utilization is improved and the high load is alleviated through frequent scheduling.

[0095] It is understandable that the scheduling cycle refers to the interval between two consecutive scheduling operations by the scheduler. In essence, the scheduling cycle is the interval between two consecutive executions of the complete scheduling process (load monitoring - weight calculation - task allocation) by the scheduler. The goal of the dynamic time slice mechanism in this application is to "reduce scheduling overhead under high load and increase task continuity under low load", as shown in Table 1.

[0096]

[0097] Table 1

[0098] In this application, the DAAP architecture faces a mixed LLM inference load (simultaneous pre-filling / decoding and long / short sequence requests) and the cycle adjustment process is as follows: If a long cycle is used under high load: the priority is not updated for a long time, which will cause newly arriving high-priority requests (such as decoding tasks for real-time user dialogues) to be unable to preempt resources in time and continue to be blocked by low-priority long tasks, which violates the core goal of DAAP to "dynamically adapt to mixed loads"; If a short cycle is used under high load: only by quickly discovering idle resources and adjusting task priorities through high-frequency scheduling can the dynamic load balancing of pre-filling and decoding in this application be achieved.

[0099] Optionally, determining the first priority of the first subtask based on the load data includes:

[0100] Based on the execution stage of the load data, a stage quantization value is determined; wherein, the stage quantization value is related to the inference characteristics of the pre-trained model;

[0101] The time priority is normalized to determine the delay value;

[0102] The resource value is determined based on the available resources of the original computing unit; wherein, the original computing unit is the computing unit corresponding to the first subtask before allocation;

[0103] The first priority is determined by weighted summation of the stage quantification value, the delay value, and the resource value.

[0104] This application sets stage quantization values ​​for each task stage of the pre-trained model based on the inference characteristics of the pre-trained model, such as the value of the decoding stage > the value of the intermediate layer > the value of the pre-filling stage. By setting stage quantization values ​​for the task stage in which each request is located, it is convenient to calculate the priority of requests in each task stage and allocate computing units. On the other hand, the time priority in this application represents the urgency of each request. For example, the larger the normalized delay value, the more urgent the request. In this application, the original computing unit is the computing unit corresponding to the first subtask before allocation, or the original computing unit is the computing unit corresponding to the second subtask before allocation, that is, the original computing unit is the computing unit corresponding to the subtask corresponding to the current request priority before allocation. This application calculates priorities and allocates computing resources through multi-parameter load data, which helps to improve the accuracy of priorities, thereby improving the accuracy and efficiency of resource allocation.

[0105] Optionally, determining the resource value based on the available resources of the original computing unit includes:

[0106] The utilization rate value is determined based on the first utilization rate of the original computing unit; wherein the utilization rate value is negatively correlated with the first utilization rate, and the first utilization rate is the collected utilization rate of the original computing unit;

[0107] The free memory value is determined based on the quotient of the free memory amount of the original computing unit and the total memory amount;

[0108] The resource value is determined by weighted summation of the utilization rate value and the idle value.

[0109] In this application, resource values ​​are determined by the utilization rate and memory status of the original computing unit, thus influencing priority values ​​based on resource availability. For example, a first utilization rate of the original computing unit is collected / detected, and a negative correlation is established between the utilization rate value and the first utilization rate; the utilization value is then determined based on the first utilization rate. Similarly, those skilled in the art can determine the relationship between the utilization rate value and the first utilization rate using a preset formula, a preset table, or a preset curve. It is understood that the amount of free memory and total memory in this application can also be obtained through collection or detection.

[0110] Optionally, the step of allocating the first subtask to the first computing unit according to the first priority includes:

[0111] If the first priority is the first identifier, the first subtask is assigned to the first-level computing unit;

[0112] Alternatively, if the first priority is the second identifier, the first subtask is assigned to the second-level computing unit; wherein the priority level represented by the first identifier is higher than the priority level represented by the second identifier, and the resource utilization rate of the first-level computing unit is lower than the resource utilization rate of the second-level computing unit.

[0113] This application utilizes an allocation strategy to determine the specific allocation logic for priorities and computing units. Specifically, high-priority requests are allocated to computing units with low resource utilization, ensuring their rapid processing. Based on the system's real-time load, this application distributes requests to idle or low-load computing units, improving resource utilization.

[0114] In some embodiments, the PAS of this application employs a multi-objective optimization algorithm for task scheduling decisions. For each arriving request... Its priority weight Calculated using the following formula:

[0115] (1-2)

[0116] Formulas 1-2 above provide the calculation formulas for the first or second priority level. Wherein, The stage of the request is (for example, pre-filling=1, decoding=3, intermediate layer=2). Delay criticality / time priority of requests (0-1 normalized, larger values ​​indicate more urgent requests); stage weights Delayed key weights Resource proportion weight The adjustment coefficients are 0.5, 0.3, and 0.2 by default. Of course, the default values ​​are just examples, and those skilled in the art can set the default values / base values ​​according to actual needs. In addition, through the dynamic adaptation of this application, the above three weights are adjusted through the "execution-evaluation-iteration" closed loop. for The percentage of GPU resources available at any given time (0-1), which is the resource value in this application. The ratio of current computing unit utilization to first utilization is given. In order to utilize numerical values, This refers to the amount of free memory. Total memory size This represents the available resource value. Of course, the above value is determined using a preset formula, and those skilled in the art can adjust it according to actual needs.

[0117] It should be noted that the value of the task stage in the request is a priority quantification indicator, not a fixed physical attribute. It can be modified to other values ​​according to the actual situation (such as pre-filling = 2, decoding = 5, intermediate layer = 3), but the relative priority of "decoding stage value > intermediate layer > pre-filling" must be maintained, that is, decoding stage ( =3) has higher priority than pre-filled ( =1), which is determined by the characteristics of LLM inference (decoding is autoregressive and interruptions will cause a sharp increase in latency; pre-filling is a one-time calculation and can be queued briefly). When modifying the value, the logic of the value must be retained. The "processing order" of LLM inference is dynamic (pre-filling and decoding are parallel under mixed loads), but the priority must be fixed to ensure decoding. In related technologies, traditional synchronous pipelines process according to the order, which causes decoding to block pre-filling. The innovation of DAAP proposed in this application is to decouple the two-stage parallel execution, but decoding has a higher priority. Therefore, the priority is distinguished by the value 3 (high) and 1 (low), instead of setting the value 1 (high) according to "pre-filling is executed first". The intermediate layer (such as temporary calculation after tensor transfer) needs to be between the two: it should not preempt the initial resources of pre-filling, nor interfere with the real-time performance of decoding. Therefore, the value is set to 2, which is in line with the priority positioning of the "transition stage".

[0118] Priority calculation is adjusted according to stage differences: Pre-filling stage (computation-intensive): Due to the need to quickly generate KVCache, the "computation resource sensitivity (α)" in the priority weight is higher (α value is increased) to ensure priority use of idle computing power; Decoding stage (memory-intensive): Due to the reliance on KVCache, the "data transfer cost (γ)" in the priority weight is higher (γ value is increased). Splitting is only allowed if the cross-unit transfer cost is lower than the local waiting cost.

[0119] In one embodiment, reference is made to Figure 2 As shown, the computing node layer of this application is "equipped with 8 A100 GPUs per node, fully interconnected via NVLink", and the "task allocation" logic of PAS is "based on the real-time load of the GPU (computing unit utilization)". Free memory The PAS algorithm allocates requests to idle or low-load GPUs, meaning the target computing unit is the GPU (including its computing cores and memory resources) to which the task is to be undertaken. Specifically, PAS first calculates the priority of all requests, and then, combined with the load status of each GPU, allocates high-priority requests to GPUs with sufficient resources (i.e., the target computing unit). For example, high-priority short-sequence decoding requests will be allocated to GPUs with Uc < 50% and Mfree > 30%, while low-priority long-sequence pre-filling requests will be allocated to the remaining idle GPUs. Of course, those skilled in the art can set allocation strategies according to actual needs, and this application does not impose specific limitations.

[0120] It should be noted that the unit in this application can be regarded as a GPU (including its computing cores and memory resources), specifically:

[0121] Single calculation unit utilization ( The terms "(device d)" and "GPU" are directly linked. For example, when the DCS module calculates the credit score, it needs to collect the "GPU computing unit utilization rate of device d". (d) In a distributed deployment, the core hardware of the computing nodes is the A100 GPU. All pre-filling / decoding tasks are executed on the GPU, and there are no other types of computing units (such as CPU computing units).

[0122] The correspondence between requests and computing units in this application is dynamic batch processing. Based on the PAS dynamic scheduling and LLM inference conventional optimization logic of this application, one GPU can process multiple low-load requests in parallel, and a high-load request can also occupy multiple GPUs (header group parallelism), as follows:

[0123] ① Load scenarios (such as multiple short sequence decoding requests): A GPU can process multiple requests in parallel through batch processing (such as processing 5 decoding tasks of 100-token at the same time). PAS will adjust the batch size according to the GPU's Uc and Mfree.

[0124] ② Load scenarios (such as long sequence pre-filled requests): A request can occupy the computing resources of multiple GPUs (such as by using header parallelism to distribute the attention head to 2 GPUs), in which case multiple GPUs jointly serve a single request;

[0125] The correspondence is dynamically adjusted by PAS based on "request priority, request complexity (token length), and GPU load", and those skilled in the art can set it according to actual needs.

[0126] Optionally, before transmitting the first data to the second computing unit via a double buffering mechanism, the method further includes:

[0127] Add metadata passport to the first data to compress the transmitted first data;

[0128] The step of transmitting the first data to the target layer of the second computing unit via a double buffering mechanism includes:

[0129] The first data of the added metadata passport is transmitted to the target layer of the second computing unit through a double buffering mechanism.

[0130] This application adds a metadata passport to the first data during inter-layer transmission. The metadata passport includes at least one of the following: target layer identifier, location index, timestamp, and inference flag. By transmitting metadata asynchronously at a fine-grained level, this application ensures that all information required for model processing is transmitted via metadata, which helps alleviate the problem of data transmission errors between layers in related technologies, thereby improving inter-layer transmission efficiency.

[0131] For example, in this application, ATP is a tensor for each transmission. Additional metadata passport :

[0132] (1-3)

[0133] in, For target layer identifier; This is the sequence position index, that is, the index of the position of the first data. For timestamps; This is a speculative flag (0 = confirmed value, 1 = speculative value, providing a basis for whether to use the prediction model in the decoding stage). The metadata passport in this application is in a defined format, including but not limited to target layer identifier, location index, and timestamp.

[0134] This speculative flag is attached by ATP during tensor transport, along with the sequence fragment / first data. Passing them together to SEB is essentially a preprocessing hint for SEB.

[0135] Optionally, before transmitting the first data to the second computing unit via a double buffering mechanism, the method further includes:

[0136] The capacity of the buffer is dynamically adjusted based on the historical transmission delay of the buffer, including:

[0137] Calculate the average transmission delay of the buffer; wherein the average transmission delay is the average of all collected transmission delays;

[0138] Obtain the first capacity and the first transmission delay of the buffer; wherein, the first capacity is the minimum designed capacity of the buffer, and the first transmission delay is the minimum transmission delay among the collected transmission delays;

[0139] The second coefficient is determined based on the quotient of the average transmission delay and the first transmission delay;

[0140] The increment value is determined based on the product of the second coefficient and the preset buffer growth step size;

[0141] The capacity of the buffer is adjusted based on the sum of the first capacity and the increase value.

[0142] This application employs a double-buffering mechanism for data transmission, ensuring that the second computing unit in the decoding stage can always obtain the required data. The buffer size B / buffer capacity is dynamically adjusted based on historical transmission delays.

[0143] (1-4)

[0144] in, Minimum buffer size / initial capacity (default 4MB); This represents the average transmission delay. The minimum measured transmission delay / the first transmission delay; Set the buffer growth step size / preset buffer growth step size (default 2MB).

[0145] Double buffering is the basic transmission architecture of the ATP communication system. Formulas 1-4 are dynamic adjustment formulas for the "buffer size" in the double buffering architecture. Both provide "architectural design" and "parameter optimization". Formulas 1-4 optimize the buffer capacity of double buffering through historical transmission delay, thereby indirectly improving the transmission efficiency of double buffering.

[0146] Specifically, the core of buffering is that two buffers work alternately to achieve parallel computation and communication; the preset formulas 1-4 are based on historical transmission delays ( , The single capacity of these two buffers can be dynamically adjusted to avoid overflow or bandwidth waste caused by fixed buffers, making the double buffer more suitable for the tensor transfer characteristics of LLM inference (such as the difference in the amount of KV Cache data with different sequence lengths).

[0147] Optionally, after the target layer performs the second sub-task based on the first data to obtain the inference result of the pre-trained model, the method further includes:

[0148] Obtain the execution data of the pre-trained model during the inference process; wherein, the execution data includes latency data, bandwidth utilization, and prediction accuracy;

[0149] Based on the execution data, update the priority calculation parameters, scheduling cycle calculation parameters, and credit value calculation parameters;

[0150] The next data to be predicted is taken as the new data to be predicted, and the acquisition step, calculation step, allocation step and reasoning step are executed sequentially based on the new data to be predicted.

[0151] The acquisition step involves: inputting the data to be predicted into a pre-trained model to obtain load data for multiple task stages of the pre-trained model; the calculation step involves: determining the first priority of the first subtask and the second priority of the second subtask based on the load data; the allocation step involves: allocating the first subtask to a first computing unit based on the first priority, so that the first subtask is executed by the first computing unit to obtain the first data; determining the second computing unit allocated to the second subtask based on the second priority; and the inference step involves: transmitting the first data to the target layer of the second computing unit through a double buffering mechanism, so that the second subtask is executed by the target layer based on the first data to obtain the inference result of the pre-trained model.

[0152] This application provides a feedback mechanism for inference optimization, feeding back execution data from the current inference optimization process to the priority calculation process, scheduling cycle calculation process, and credit value calculation process. This updates the priority calculation parameters, scheduling cycle calculation parameters, and credit value calculation parameters, allowing for new inference optimization to be performed in the next inference optimization process using these updated parameters. This feedback mechanism improves the accuracy of inference optimization by updating parameters in real time, thereby enhancing resource utilization.

[0153] Optionally, if the matching probability between the candidate sequence and the first data is greater than or equal to a preset probability threshold, before using the candidate sequence as the inference result, the method further includes:

[0154] Obtain a basic threshold and a second utilization rate and a third utilization rate of the second computing unit; wherein, the second utilization rate is the collected utilization rate of the second computing unit, and the third utilization rate is the maximum allowable utilization rate of the second computing unit;

[0155] A third coefficient is determined based on the second utilization rate and the third utilization rate; wherein the third coefficient is negatively correlated with the second utilization rate.

[0156] The preset probability threshold is determined based on the product of the basic threshold and the third coefficient.

[0157] This application adjusts the preset probability threshold / verification threshold by utilizing the utilization rate of the second computing unit / system load. A preset probability threshold determines the comparison value for accepting the inferred result:

[0158] (1-5)

[0159] in, Set as the base threshold / basic threshold (default 0.7); The ratio of current computing unit utilization to second utilization. Maximum permissible utilization / third utilization; It is the third coefficient.

[0160] Specifically, when When (computing unit utilization) is high, the system is under high load (e.g., GPU core utilization is close to 100%). At this point, designing a low threshold serves two key purposes:

[0161] ① Reduce redundant calculations and alleviate load pressure: Under high load, if the threshold is too high (strict requirements on candidate sequences), a large number of inference results will be rejected, and the entire decoding process needs to be re-executed. This will consume additional computing resources that are already strained, further exacerbating GPU congestion and increasing latency. Lowering the threshold can improve the "pass rate" of candidate sequences, reduce retries, and allow the system to process existing tasks more efficiently, avoiding a vicious cycle of load.

[0162] ② Prioritize throughput, moderately sacrificing accuracy tolerance: In high-load scenarios (such as multiple requests queuing under mixed load), the core requirement of LLM inference is to "process more tasks as quickly as possible" (prioritizing throughput), rather than maximizing inference accuracy. Although a low threshold may allow a small number of low-probability candidate sequences to pass, it results in shorter decoding time, faster task flow, and higher overall system efficiency.

[0163] 2. When When the system has a large amount of idle computing resources (e.g., GPU core utilization is only 30%), formula (1-5) will generate a high threshold:

[0164] ① Utilize idle resources to ensure accuracy and reduce retry waste: Under low load, the system has enough resources to strictly verify the inference results (high threshold), which can filter out low-probability candidate sequences and reduce repeated decoding caused by inference errors. At this time, even if the verification process takes a little longer, it will not affect the overall latency. On the contrary, it can avoid idle resources being occupied by invalid retries.

[0165] ② Balancing acceleration benefits with resource waste: Under low load, the urgency for speculation acceleration decreases, making it even more important to avoid introducing errors for the sake of acceleration, which would waste idle resources. A high threshold makes speculation results more reliable, ensuring that the acceleration benefits of SEB outweigh the cost of errors.

[0166] Optionally, before using the candidate sequence as the inference result if the matching probability between the candidate sequence and the first data is greater than or equal to a preset probability threshold, the method further includes:

[0167] The first data is subjected to feature embedding processing to obtain a first feature; the candidate sequence is subjected to feature embedding processing to obtain a second feature;

[0168] The first feature and the second feature are spatially aligned to obtain the third feature;

[0169] The third feature is subjected to probability normalization to obtain the matching probability.

[0170] The matching probability of the candidate sequence in this application is calculated using the following formula:

[0171] (1-6)

[0172] in, For embedded functions; The learned projection matrix (128×128); Input sequence / first data; These are candidate sequences.

[0173] Formulas 1-6 (candidate sequence probability calculation formulas) are the core quantization logic of the speculative execution buffer (SEB). The principle is to accurately calculate the matching reliability between the candidate sequence generated by the lightweight model and the input sequence through three steps: feature embedding, spatial alignment, and probability normalization, so as to provide a quantitative basis for subsequent adaptive threshold verification.

[0174] Optionally, before inputting the first data into the prediction model to predict the decoded output and obtain the candidate sequence, the method further includes:

[0175] Obtain the speculative identifier of the first data transmitted to the second computing unit;

[0176] If the inferred identifier represents an inferred value, the first data is input into the prediction model to obtain a candidate sequence;

[0177] Alternatively, if the inferred identifier is a definite value, a decoding task is performed based on the first data to obtain the inference result.

[0178] This application determines the speculation identifier through a speculation flag in the metadata. The speculation identifier is used to characterize whether the inference result of the decryption subtask is obtained through the prediction model. In some embodiments, 1 can be used to represent the speculation value and 0 can be used to represent the certain value. Of course, after calculating the matching probability of the candidate sequences obtained by the prediction model, this application determines whether to use the candidate sequences predicted by the prediction model as the final decoding inference result through the result source identifier. This application plans and verifies the decoding process through the speculation identifier and the result source identifier, and reduces the resource consumption of the decoding process through a lightweight model.

[0179] Optionally, after the target layer performs the second subtask based on the first data to obtain the inference result of the pre-trained model, the method further includes:

[0180] Obtain operational data for each task stage;

[0181] Based on the operational data, a credit value is determined, and the system's processing rate is adjusted accordingly.

[0182] This application collects runtime data during the execution of each task stage of the pre-trained model, determines a credit value based on the runtime data, and then adjusts the system's processing rate. Specifically, the credit value characterizes the system's computational resource utilization efficiency, and the processing rate of each computing unit in the system is adjusted by adjusting the credit value to achieve a dynamic balance of global resources.

[0183] Optionally, the operational data includes the system's free memory, total memory, and fourth utilization rate, and determining the credit value based on the operational data includes:

[0184] The fourth coefficient is determined based on the quotient of the free memory and the total memory;

[0185] A first value is determined based on the fourth utilization rate and the utilization rate threshold; wherein the utilization rate threshold is a preset maximum acceptable utilization rate of the system;

[0186] The credit value is determined based on the product of the first value and the fourth coefficient.

[0187] The system of this application includes a first computing unit and / or a second computing unit. The utilization threshold of this application is an upper limit warning value (default 0.8, i.e., 80%). This threshold is the standard for judging whether the equipment is overloaded. For example, when the fourth utilization rate is less than or equal to the utilization threshold, the equipment resources are sufficient, the equipment credit value is high, and it can undertake more tasks; when the fourth utilization rate is greater than the utilization threshold, the equipment is close to full load, the equipment credit value drops sharply, and the processing rate needs to be reduced through formula (1-8) to avoid resource exhaustion.

[0188] Those skilled in the art can adjust the utilization threshold based on the following factors: the performance ceiling of the hardware device, which needs to match the actual computing power bottleneck of the computing unit to avoid the threshold exceeding the hardware's stable operating threshold and causing a crash; the resource requirements of the task type, distinguishing between pre-filling (computation-intensive) and decoding (memory-intensive) load characteristics to ensure that the utilization threshold can cover the peak load of dual tasks; and the system latency and throughput targets, where the threshold needs to balance high utilization (improving throughput) with low latency (avoiding blocking). Real-time interactive scenarios (such as chatbots) require a lower threshold to reserve resources, while non-real-time scenarios (such as batch inference) can have a higher threshold.

[0189] Understandably, during the execution of the pre-trained model, the validation pass rate and decoding time reflect the efficiency of current computing resource utilization (e.g., a high pass rate means low resource consumption), and are used to correct the device credit value. Each device Credit score The calculation is as follows:

[0190] (1-7)

[0191] in, For equipment Free memory; For equipment Total memory; For equipment Computational unit utilization / Fourth utilization; This is the utilization threshold (default 0.8).

[0192] It should be noted that the device in DCS specifically refers to a single GPU, while a computing node is a collection of multiple GPUs. DCS achieves finer-grained resource balancing through GPU-level credit value calculation.

[0193] Optionally, adjusting the system's processing rate based on the credit value includes:

[0194] Determine the amount of credit change based on continuous credit scores;

[0195] The second value is determined based on the product of the credit change and the preset adjustment coefficient;

[0196] Based on the second value and the first processing rate, a new processing rate is determined, and the system's processing rate is adjusted to the new processing rate; wherein, the first processing rate is the collected system processing rate.

[0197] This application adjusts the system's processing rate based on changes in credit value. The system here refers to a first processing unit and / or a second processing unit. It is understood that fluctuations in computing unit utilization (such as a sudden increase in GPU load due to speculation failure) will affect the change in credit value. The calculation is used to adjust the processing rate. (When utilization rate increases sharply) If the value is negative, the rate is reduced to alleviate the pressure. In this application, credit values ​​are exchanged between devices via a distributed ledger, and the adjustment strategy is as follows:

[0198] (1-8)

[0199] in, For processing speed; The processing rate used by the GPU before the rate adjustment refers to the first processing rate in this application. λ The preset adjustment coefficient (default 0.1); ΔC This represents the change in creditworthiness. For example, in the decoding process, if the inference accuracy is high and the decoding time is short, it indicates that the current device resource pressure is relatively low, Δ... C If the value is positive, the processing speed can be increased appropriately; if the prediction fails frequently and repeated calculations are required, then Δ C If the value is negative, the rate needs to be reduced to alleviate resource constraints.

[0200] Processing speed in this application r The rate at which a single GPU compute unit performs pre-filling / decoding tasks (i.e., the efficiency metric for GPUs processing LLM inference tasks) is adjusted by logic entirely dependent on the change in device credit value Δ. C This enables dynamic balancing of global resources.

[0201] Optionally, before adjusting the system's processing rate based on the credit value, the method further includes:

[0202] If the credit change corresponding to the credit value is greater than or equal to the preset credit threshold, the system processing rate is adjusted according to the credit value.

[0203] Alternatively, if the credit change corresponding to the credit value is less than a preset credit threshold, the system's processing rate is maintained; wherein, the preset credit threshold is the credit change corresponding to a significant change in resources.

[0204] In this application, the transfer of credit values ​​between nodes follows a process of credit value calculation, rate adjustment, and global synchronization, and employs an incremental update mechanism where the change exceeds a preset credit threshold. Specifically, the trigger condition for credit value transfer is that only credit values ​​exceeding the preset credit threshold are transmitted. The incremental update mechanism ensures that nodes synchronize credit information via the distributed ledger only when the change in credit value (ΔCredit) exceeds the preset credit threshold. For example, if node A's Credit(d) changes from 0.8 to 0.85 (a 6.25% change), transmission is triggered, and the processing rate of the computing unit is adjusted using formulas 1-8. This application avoids invalid communication caused by high-frequency, small fluctuations (such as a 1% instantaneous fluctuation) through the incremental update mechanism, while ensuring that significant changes in resource status (such as a sudden increase in GPU load causing a sharp drop in Credit(d)) are detected in a timely manner.

[0205] This application provides global credit synchronization, where incrementally transmitted credit values ​​are directly used as input for other nodes to calculate ΔC. For example, if node B receives a Credit(d) update (change of 6%) from node A, then after updating its local credit ledger, node B recalculates its coordination rate with A (e.g., if A's Credit decreases, B reduces the rate at which it sends tasks to A).

[0206] Optionally, the first data includes at least one of a first tensor, a second tensor, and a third tensor, and the step of transmitting the first data to the second computing unit through a double buffering mechanism includes:

[0207] In the first cycle, the first tensor is loaded through the first buffer, the second tensor is transmitted through the second buffer, and the second subtask is executed by the second computing unit using the second tensor transmitted through the second buffer.

[0208] In the second cycle, the first tensor is transmitted through the first buffer, the next batch of the third tensor is loaded through the second buffer, and the second subtask is executed by the second computing unit using the first tensor transmitted through the first buffer; wherein the first cycle and the second cycle are two consecutive cycles.

[0209] This application improves the efficiency of inter-layer data transmission through a double buffering mechanism. The specific process is as follows (taking the transmission of tensors from the pre-filling stage to the decoding stage as an example):

[0210] 1. First cycle:

[0211] ① Buffer A: Loads tensor data generated during the pre-filling phase (such as KV Cache fragment \(S{1:k}\));

[0212] ②Buffer B: Transmits the loaded tensor data to the target layer in the decoding stage via NVLink / Infinity Fabric;

[0213] ③ Decoding Calculation Unit: Performs decoding operations using the data that has been transmitted in buffer B;

[0214] In this application, the three operations are performed in parallel, and the computing unit does not need to wait for the "data loading + transmission" to be completed, thus realizing "communication without blocking computing".

[0215] 2. Second cycle:

[0216] ① Buffer A: Transfers newly generated tensor data to the target layer (role switches to "transfer buffer");

[0217] ② Buffer B: Load the next batch of tensor data (switching roles to "Load Buffer");

[0218] ③ Decoding Calculation Unit: Continues decoding using the data that has already been transmitted in buffer A;

[0219] The two buffers take turns loading and transferring, forming a continuous parallel pipeline that maximizes hardware bandwidth utilization.

[0220] The inference optimization method for the pre-trained model provided in this application will be described in detail below with specific embodiments:

[0221] Figure 2 The core function of the GPU control unit is to coordinate the processing order of various functional modules (SEB, ATP interface, DCS monitoring unit) on the local GPU, ensuring that computational tasks (pre-filling / decoding) are synchronized with data transmission, speculative execution, and resource monitoring. Specifically, this includes: coordinating data transmission (ATP) and computation: controlling the ATP interface to receive scheduling instructions from PAS, prioritizing the transmission of high-priority requested tensor data (with passport metadata), ensuring that the GPU computing core is ready when the data arrives. Coordinating speculative execution (SEB) and decoding: controlling the execution timing of the SEB buffer—before the GPU executes the current decoding step, first starting the SEB lightweight model to generate candidate sequences; if the verification passes, the results are directly reused, avoiding GPU core waiting. Coordinating resource monitoring (DCS) and task execution: receiving credit value data from the DCS monitoring unit in real time; if the GPU's... If the load exceeds 80% (high load), new requests will be paused, and already assigned tasks will be completed first to prevent overload. Essentially, it is a local "module coordination hub" on the GPU, ensuring a logical closed loop (load monitoring - task execution - result feedback) for the four major DAAP modules at the single GPU level.

[0222] These modules together form a complete inference acceleration system that can automatically adjust its operating strategy based on the actual load, maximizing resource utilization efficiency while ensuring service quality.

[0223] Reference Figure 3 The overall technical approach is as follows: Step 1 (PAS module) generates a request priority queue through task scheduling (processing priority of each request, target computing unit, sequence index). k Step 2 (ATP module) calculates the sequence fragments transmitted non-blockingly based on the scheduling results of Step 1. (Includes tensor passport metadata, including sequence position, timestamps, etc.); Step 3 (SEB module) is based on Step 2. The system obtains the inference execution results (including candidate sequence verification pass rate, decoding time, and computing unit utilization fluctuation) through a lightweight model. Step 4 combines the scheduling load (computing unit utilization, free memory, and total memory) from Step 1 with the inference execution results from Step 3 to adjust resource allocation. Step 5 integrates the final decoded sequence from Step 3 with the resource adjustment records from Step 4 to generate a user-understandable result and return it (where the prediction model can be the transformer layers in this application). This completes the current task interaction. Then, the system collects key data from this inference (such as latency, bandwidth utilization, and inference accuracy) and feeds it back to Step 1 (PAS) and Step 4 (DCS) to optimize the priority weights, scheduling cycle, and credit value calculation parameters for the next round of tasks. This ultimately forms a closed loop of "execution-evaluation-iteration" to ensure that the system dynamically adapts to load changes during continuous operation and continuously improves resource utilization and inference efficiency.

[0224] Step 1: Priority Adaptive Scheduler (PAS);

[0225] Step 1's technical approach is a cyclical process of load monitoring, weight calculation, dynamic scheduling cycle adjustment, and task allocation: Load data of pre-filled / decoded tasks is collected in real time, substituted into a multi-dimensional weight formula to calculate priorities, the scheduling cycle is adjusted according to load differences, and finally, task allocation is completed based on priorities (including the processing priority of each request, target computing unit, and sequence index). The target computing unit is the GPU (and associated resources) allocated by PAS for the task to be executed based on the load monitoring results. Figure 4 As shown. The priority adaptive scheduler calculates request priorities using Formula 1-2 and allocates requests according to these priorities; it adjusts the scheduler's scheduling cycle using Formula 1-1 to adapt to changes in system load.

[0226] Step 2: Asynchronous Tensor Transfer (ATP) and Buffer Optimization;

[0227] ATP determines the target layer for tensor transport based on the target computational unit in step 1. d Through sequence indexing k Mark the sequence fragment to be transmitted Based on the priority ranking, transmission bandwidth resources are allocated to high-priority requested tensor data to ensure their priority transmission. The technical approach is tensor passport generation - double-buffered transmission - dynamic buffer adjustment: Metadata passports are attached to the tensors to be transmitted (intermediate sequence data generated during the pre-filling stage, i.e., inter-layer data), non-blocking transmission is achieved through a double-buffering mechanism, and the buffer size is dynamically adjusted according to the transmission delay. Figure 5 As shown.

[0228] ATP not only handles the transfer of the target layer across other nodes (cross-GPU), but also covers tensor transfers across all scenarios, including within a GPU, across GPUs within a node, and across GPUs across nodes. The core function of ATP is to connect the data transmission channels of various computing units, decoupling computation and communication, and deeply integrating with underlying hardware (such as NLink or Infinity Fabric). These computing units include: different model layers within a single GPU (e.g., after the KV Cache is generated in the pre-filling stage, it needs to be transferred to the Transformer layer in the decoding stage), i.e., the first and second computing units are the same computing unit; different GPUs within the same node (e.g., 8 A100 GPUs within a computing node interconnected via NVLink, tensor transfer of attention heads when the head group is parallel); GPUs between different nodes (e.g., long sequence tensor transfers across computing nodes); and cases where the first and second computing units are not the same computing unit.

[0229] Asynchronous tensor transfer adds metadata passports to the tensors being transferred using formula 1-3 to refine the data being transferred; the double buffering mechanism adjusts the size of each buffer using formula 1-4 to improve the transfer efficiency of double buffering.

[0230] Step 3: Speculate Execution Buffer (SEB);

[0231] In the asynchronous tensor transfer phase of step 2, the sequence data generated in the pre-filling phase (including...) The corresponding feature tensor will be transmitted to the decoding stage through the non-blocking channel of the ATP system. At this time, the sequence position index in the tensor passport metadata... k The length information of the sequence will be marked to ensure... The sequence is accurately routed to the Speculative Execution Buffer (SEB), and the speculative flag f indicates to the SEB whether the sequence is a speculative value, assisting subsequent verification logic (such as using a higher verification threshold for speculative values). The task allocation result of the Priority Adaptive Scheduler (PAS) in step 1 will determine the sequence's validity. The transmission priority and target computing unit are determined to ensure that SEB can acquire the required sequence data in a timely manner for candidate prediction.

[0232] Step 3's technical approach is candidate sequence generation - probability calculation - adaptive verification - result output; the judgment process involves inputting the sequence... Through lightweight models (6-layer Transformer, 128 hidden dimensions) predicts and generates candidate sequences (the candidate sequences are the decoded outputs of the lightweight model's predictions). Their probabilities are calculated using formulas 1-6 and compared with an adaptive threshold / probability threshold. If the verification passes, the candidate result is output; otherwise, normal decoding is performed. The inference execution results include inference execution efficiency (with the candidate sequence verification pass rate as the core indicator). (The proportion of the model) reflects the lightweight model. The accuracy of predictions), decoding time (the actual time to complete a single decoding step (including "acceleration time" when the prediction verification is successful and "retry time" when the verification fails); the shorter the time, the higher the execution efficiency), and fluctuation in computing unit utilization (the proportion of repeated calculations caused by prediction failures (such as the computing power consumption of re-executing the decoding step after a prediction error); the lower the waste rate, the higher the execution efficiency). Figure 6 As shown.

[0233] This application hypothesizes that the marker is attached by ATP during tensor transport and accompanies the sequence fragment. Passing them together to SEB is essentially a preprocessing hint for SEB, allowing them to infer the flag bits. f The initial screening condition is not sufficient to determine the final decoding result and its source on its own. It needs to be combined with SEB's candidate generation-probability verification-result selection process to form a complete judgment logic, which is divided into two steps:

[0234] 1. First step: Determine the flag position ( f This determines the processing logic of SEB (whether to initiate speculative verification).

[0235] ① If the input is a tensor of SEB correspond f =1 (speculated value): SEB will activate a lightweight model to generate candidate sequences; And calculate the probability according to the "higher verification threshold" (i.e. Formula 1-6) and enter the verification stage. At this time, the flag indicates that the predicted value needs to be verified, but it is not yet determined whether the predicted result will be adopted in the end.

[0236] ②If the input is a tensor of SEB correspond f =0 (determined value): This tensor is the basic data for actual calculation (such as pre-filled KV cache). SEB does not need to generate candidate sequences, but directly passes it as the decoding input to the LLM main model. At this time, the flag indicates that there is no need for speculation and verification, and the decoding result must be the actual calculated value.

[0237] 2. Second step: The SEB verifies the result to determine the final decoding result and its source, and updates the meaning of the flag bits. The SEB uses a dynamic threshold θ (Formula 1-5) to determine whether to accept the speculative value. The source of the final decoding result and its corresponding flag bits need to be determined in conjunction with the verification result. Therefore, relying solely on speculative flag bits ( f The final decoding result and its source cannot be determined independently. It is necessary to first determine whether to start verification through f, and then determine the source of the result through the verification result. The final decoding result will carry a "flag bit that matches the source" (presumably 1 for success, 0 for actual decoding).

[0238] Step 4: Credit score calculation and resource adjustment (DCS module).

[0239] Step 4 employs a distributed process of credit score calculation, rate adjustment, and global synchronization, based on the load data from Step 1 (computing unit utilization). Free memory Total memory The execution results of step 3 and step 4 are used to calculate the credit value of each device, adjust the processing rate accordingly, and synchronize global credit information. Figure 7 As shown.

[0240] Step 5: Output Results and System Feedback.

[0241] Reference Figure 8 As shown, step 5 integrates the final decoding sequence from step 3 with the resource adjustment record from step 4 to generate a user-understandable result and return it, completing the current task interaction. Then, it collects key data from this inference (such as latency, bandwidth utilization, and inference accuracy) and feeds it back to step 1 (PAS) and step 4 (DCS) respectively to optimize the priority weight, scheduling cycle, and credit value calculation parameters of the next round of tasks. Finally, it forms a closed loop of execution-evaluation-iteration, ensuring that the system dynamically adapts to load changes during continuous operation and continuously improves resource utilization and inference efficiency.

[0242] Specifically, although the formulas (1-1, 1-2) in step 1 do not directly calculate latency, bandwidth utilization, and prediction accuracy, the three are deeply related through an indirect link of "scheduling logic - resource allocation - performance metrics" (step 1 is the "source scheduling layer" for performance metrics optimization); while formula 1-4 (dynamic adjustment of buffer) in step 2 is a latency optimization method in the "direct execution layer".

[0243] Distributed Multi-Node Deployment Scheme: This embodiment describes the deployment method of the DAAP system in a distributed computing environment. The system adopts a layered architecture design, such as... Figure 9 As shown, where:

[0244] (1) Computation Node Layer: Each node is equipped with 8 A100 GPUs, interconnected via a high-speed fully interconnected switch (such as NVLink fully interconnected), running an independent DAAP instance. A head-group parallel strategy is adopted within the node to evenly distribute attention heads across the GPUs. NVLink fully interconnected is used for low-latency interconnection between the 8 A100 GPUs. The cross-node asynchronous tensor passport module is used to compress passport metadata data transmission. Specifically, the distributed speculative execution buffer module in compute node 1 functions as a predictive model parameter sharing mechanism and is also used for All Gather synchronization. The dynamic credit system incremental update module functions as a module that only transmits >5% of credit changes. A100 GPU1 is head-group parallel with attention shard 1. The local dynamic adaptive asynchronous pipeline instance includes the PAS module, ATP module, SEB module, and DCS module, implementing the functions of this application. The tensor core, also known as the CUDA core, performs pre-filled decoding computation. The CPU memory stores KV cache shards and intermediate tensors. A100 GPU2 is head-group parallel with attention shard 2. The remote direct memory access network between the compute node layer and the coordination node layer includes RDMA networks, and the high-speed storage networks connecting the compute node layer and the storage node layer include PCIe 5.0 and NVMe-oF.

[0245] (2) Coordination Node Layer: Deploy a central scheduling server (hardware core includes CPU and memory) and run an enhanced PAS module. This module extends the cross-node credit exchange protocol, supports global resource view synchronization based on RDMA, and has an update cycle of 10ms. The update cycle is the "time interval for the PAS module to synchronize cross-node global resource status data," that is, PAS collects resource data from each computing node and updates the "global resource view" every 10ms, providing a real-time basis for dynamic scheduling. High-speed network cards include RDMA high-speed network cards, which have the characteristics of supporting global resource view synchronization and an update cycle of 10ms.

[0246] (3) Storage Node Layer: Configure a high-speed NVMe storage cluster to cache the KV cache during the pre-filling stage. A sharded storage strategy is adopted, with each compute node responsible for managing the cache partition of its processing sequence. Specifically, the high-speed storage network card supports PCIe 5.0 and NVMe-oF for low-latency access. The key-value cache shard controller's function is to divide the cache partitions by compute node and manage read and write operations. The high-speed storage cluster hardware consists of multiple NVMe SSDs forming a RAID array.

[0247] This application offers dynamic resource management: through the collaborative work of a priority adaptive scheduler and a dynamic credit system, real-time optimal allocation of computing resources is achieved; breakthroughs in communication efficiency: the asynchronous tensor passport protocol reduces inter-layer communication overhead through metadata management and a double-buffering mechanism. In distributed deployments, this technology is particularly suitable for long-sequence transmission scenarios across nodes; innovative latency optimization: the speculative execution buffer combined with a load-aware verification mechanism reduces decoding latency while maintaining prediction accuracy, a technology of significant value for real-time interactive applications; system scalability: the DAAP architecture can flexibly adapt to different hardware environments (from data centers to edge devices) and task scenarios (from unimodal to multimodal). Modular design supports seamless integration with future new accelerators and model architectures.

[0248] This application proposes a dynamic priority scheduling mechanism: unlike existing static batch processing or single-dimensional priority scheduling, it pioneers a variable time-slice EDF algorithm based on multi-dimensional weights (stage, delay, resources), which can adapt to task priority adjustments under mixed loads in real time. It designs an asynchronous tensor passport protocol: breaking through the limitations of traditional synchronous communication or simplified asynchronous transmission, it solves the problems of data transmission errors and overflows between layers through fine-grained asynchronous transmission with metadata (target layer, sequence index, timestamp, etc.) and a double-buffered dynamic adjustment mechanism. It develops a load-aware speculative execution scheme: unlike fixed-threshold speculative mechanisms, it adopts a lightweight prediction model combined with a verification threshold dynamically adjusted according to computational utilization, balancing prediction accuracy and computational overhead under high load and fully utilizing resources for acceleration under low load. It constructs a distributed credit adjustment system: existing technologies lack global resource coordination; this invention achieves dynamic resource balancing across devices by quantifying device credit values ​​based on memory and computational pressure, solving the problem of uneven load across different devices.

[0249] It is understood that the pre-trained model in this application can be computer vision (CV) inference: applying the DAAP architecture to video frame generation (such as diffusion model inference), with the pre-filling stage corresponding to image feature extraction and the decoding stage corresponding to frame sequence generation. Feature tensors are transmitted via ATP, and the SEB predicts candidates for the next frame, improving the efficiency of real-time video generation. It can also be an edge AI device: adapting to embedded scenarios (such as smart car cockpits), simplifying the SEB to a single-layer Transformer, and dynamically adjusting CPU / GPU resource allocation through DCS to support low-power inference for models with 7 billion parameters. Furthermore, it can be a multimodal large model: extending the ATP protocol to support cross-modal tensor (text, image, voice) transmission, with DCS allocating independent credit channels for different modal tasks, improving the throughput of multimodal generation (such as mixed text and image reports).

[0250] For different task scenarios such as computer vision, edge AI, and multimodal computing, the model processing is first divided into different stages based on the scenario's "hardware constraints (such as low computing power at the edge), load characteristics (such as time-series frame generation in CV and cross-modal data in multimodal computing), and efficiency goals (such as real-time performance / low power consumption / high throughput)" (model parallel allocation). Then, through scenario-based modifications to the DAAP core module (such as SEB lightweighting, ATP cross-modal transmission, and DCS modal credit channel), "dynamic collaborative parallelism" is achieved in the divided stages (PAS scheduling priority, ATP data synchronization, SEB accelerated execution, and DCS resource balancing), ultimately solving the inference efficiency bottlenecks in different scenarios (such as CV real-time performance, low power consumption at the edge, and multimodal throughput).

[0251] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0252] Embodiments of this application also provide an inference optimization apparatus for a pre-trained model, comprising:

[0253] The acquisition module is used to input the data to be predicted into the pre-trained model and acquire the load data of multiple task stages of the pre-trained model; wherein, the load data includes the current task stage, available resources, and the time priority of the task stage; the task stage includes a first subtask and a second subtask.

[0254] The calculation module is used to determine the first priority of the first subtask and the second priority of the second subtask based on the load data;

[0255] The allocation module is configured to allocate the first subtask to the first computing unit according to the first priority, so that the first subtask can be executed by the first computing unit to obtain the first data;

[0256] The inference module is used to determine the second computing unit to be allocated to the second subtask according to the second priority; and to transmit the first data to the target layer of the second computing unit through a double buffering mechanism, so as to execute the second subtask based on the first data in the target layer to obtain the inference result of the pre-trained model.

[0257] For a description of the features in the embodiment corresponding to the inference optimization device of the pre-trained model, please refer to the relevant description of the embodiment corresponding to the inference optimization method of the pre-trained model, which will not be repeated here.

[0258] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in the inference optimization method embodiments of any of the pre-trained models described above.

[0259] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the inference optimization method embodiments of any of the pre-trained models described above when running.

[0260] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0261] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the inference optimization method embodiments of any of the pre-trained models described above.

[0262] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the inference optimization method embodiments of any of the pre-trained models described above.

[0263] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0264] The foregoing has provided a detailed description of the inference optimization method and electronic device for a pre-trained model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for optimizing inference of a pre-trained model, characterized in that, The method includes: The data to be predicted is input into the pre-trained model to obtain the load data of multiple task stages of the pre-trained model; wherein, the load data includes the task stage, available resources, and time priority of the task stage; the task stage includes a first sub-task and a second sub-task; the data to be predicted includes text data or video data; Based on the load data, determine the first priority of the first subtask and the second priority of the second subtask; According to the first priority, the first subtask is assigned to the first computing unit so that the first subtask is executed by the first computing unit to obtain the first data; Based on the second priority, a second computing unit is determined to be assigned to the second subtask; the first data is transmitted to the target layer of the second computing unit through a double buffering mechanism, so that the second subtask is executed on the target layer based on the first data to obtain the inference result of the pre-trained model; Based on the first data, the second subtask is performed to obtain the inference result of the pre-trained model, including: The first data is input into the prediction model to predict the decoded output, thus obtaining the candidate sequence; If the matching probability between the candidate sequence and the first data is greater than or equal to a preset probability threshold, the candidate sequence is taken as the inference result; Alternatively, if the matching probability between the candidate sequence and the first data is less than a preset probability threshold, a decoding task is performed based on the first data to obtain the inference result; wherein, the preset probability threshold is the minimum acceptable prediction accuracy of the prediction model; Before transmitting the first data to the second computing unit via a double buffering mechanism, the method further includes: Add metadata passport to the first data to compress the transmitted first data; The step of transmitting the first data to the target layer of the second computing unit via a double buffering mechanism includes: The first data of the added metadata passport is transmitted to the target layer of the second computing unit through a double buffering mechanism.

2. The inference optimization method for the pre-trained model according to claim 1, characterized in that, After allocating the first subtask to the first computing unit according to the first priority, the method further includes: Obtain the cumulative duration; wherein, the cumulative duration is the time elapsed since the last resource scheduling; If the cumulative duration is greater than or equal to the scheduling period, the cumulative duration is cleared to zero, and the load data of multiple task stages of the pre-trained model is returned to determine the priority based on the load data so as to reschedule the computing resources; wherein, the scheduling period is the preset duration between two consecutive resource scheduling.

3. The inference optimization method for the pre-trained model according to claim 2, characterized in that, Before clearing the accumulated duration to zero if the accumulated duration is greater than or equal to the scheduling period, the method further includes: The base duration is determined based on the earliest deadline priority algorithm; Obtain the average load of all computing units, and determine a first coefficient based on the first load and the average load; wherein the first coefficient is negatively correlated with the average load, and the first load is the maximum allowable load of the system; The scheduling period is determined based on the product of the first coefficient and the base duration.

4. The inference optimization method for the pre-trained model according to claim 2, characterized in that, Before clearing the accumulated duration to zero if the accumulated duration is greater than or equal to the scheduling period, the method further includes: If the system load is the first load, the scheduling period is determined to be the first duration; Alternatively, if the system load is the second load, the scheduling period is determined to be the second duration; wherein the first load is higher than the second load, and the first duration is less than the second duration.

5. The inference optimization method for the pre-trained model according to claim 1, characterized in that, Based on the load data, determine the first priority of the first subtask, including: Based on the execution stage of the load data, a stage quantization value is determined; wherein, the stage quantization value is related to the inference characteristics of the pre-trained model; The time priority is normalized to determine the delay value; The resource value is determined based on the available resources of the original computing unit; wherein, the original computing unit is the computing unit corresponding to the first subtask before allocation; The first priority is determined by weighted summation of the stage quantification value, the delay value, and the resource value.

6. The inference optimization method for the pre-trained model according to claim 5, characterized in that, Determining resource values ​​based on the available resources of the original computing unit includes: The utilization rate value is determined based on the first utilization rate of the original computing unit; wherein the utilization rate value is negatively correlated with the first utilization rate, and the first utilization rate is the collected utilization rate of the original computing unit; The free memory value is determined based on the quotient of the free memory amount of the original computing unit and the total memory amount; The resource value is determined by weighted summation of the utilization rate value and the idle value.

7. The inference optimization method for the pre-trained model according to claim 1, characterized in that, The step of allocating the first subtask to the first computing unit according to the first priority includes: If the first priority is the first identifier, the first subtask is assigned to the first-level computing unit; Alternatively, if the first priority is the second identifier, the first subtask is assigned to the second-level computing unit; wherein the priority level represented by the first identifier is higher than the priority level represented by the second identifier, and the resource utilization rate of the first-level computing unit is lower than the resource utilization rate of the second-level computing unit.

8. The inference optimization method for the pre-trained model according to claim 1, characterized in that, Before transmitting the first data to the second computing unit via a double buffering mechanism, the method further includes: The capacity of the buffer is dynamically adjusted based on the historical transmission delay of the buffer, including: Calculate the average transmission delay of the buffer; wherein the average transmission delay is the average of all collected transmission delays; Obtain the first capacity and the first transmission delay of the buffer; wherein, the first capacity is the minimum designed capacity of the buffer, and the first transmission delay is the minimum transmission delay among the collected transmission delays; The second coefficient is determined based on the quotient of the average transmission delay and the first transmission delay; The increment value is determined based on the product of the second coefficient and the preset buffer growth step size; The capacity of the buffer is adjusted based on the sum of the first capacity and the increase value.

9. The inference optimization method for the pre-trained model according to claim 1, characterized in that, After the second subtask is performed at the target layer based on the first data to obtain the inference result of the pre-trained model, the method further includes: Obtain the execution data of the pre-trained model during the inference process; wherein, the execution data includes latency data, bandwidth utilization, and prediction accuracy; Based on the execution data, update the priority calculation parameters, scheduling cycle calculation parameters, and credit value calculation parameters; The next data to be predicted is taken as the new data to be predicted, and the acquisition step, calculation step, allocation step and reasoning step are executed sequentially based on the new data to be predicted. The acquisition step involves: inputting the data to be predicted into a pre-trained model to obtain load data for multiple task stages of the pre-trained model; the calculation step involves: determining the first priority of the first subtask and the second priority of the second subtask based on the load data; the allocation step involves: allocating the first subtask to a first computing unit based on the first priority, so that the first subtask is executed by the first computing unit to obtain the first data; determining the second computing unit allocated to the second subtask based on the second priority; and the inference step involves: transmitting the first data to the target layer of the second computing unit through a double buffering mechanism, so that the second subtask is executed by the target layer based on the first data to obtain the inference result of the pre-trained model.

10. The inference optimization method for the pre-trained model according to claim 1, characterized in that, If the matching probability between the candidate sequence and the first data is greater than or equal to a preset probability threshold, before using the candidate sequence as the inference result, the method further includes: Obtain a basic threshold and a second utilization rate and a third utilization rate of the second computing unit; wherein, the second utilization rate is the collected utilization rate of the second computing unit, and the third utilization rate is the maximum allowable utilization rate of the second computing unit; A third coefficient is determined based on the second utilization rate and the third utilization rate; wherein the third coefficient is negatively correlated with the second utilization rate. The preset probability threshold is determined based on the product of the basic threshold and the third coefficient.

11. The inference optimization method for the pre-trained model according to claim 1, characterized in that, Before using the candidate sequence as the inference result if the matching probability between the candidate sequence and the first data is greater than or equal to a preset probability threshold, the method further includes: The first data is subjected to feature embedding processing to obtain a first feature; the candidate sequence is subjected to feature embedding processing to obtain a second feature; The first feature and the second feature are spatially aligned to obtain the third feature; The third feature is subjected to probability normalization to obtain the matching probability.

12. The inference optimization method for the pre-trained model according to claim 1, characterized in that, Before inputting the first data into the prediction model to predict the decoded output and obtain the candidate sequence, the method further includes: Obtain the speculative identifier of the first data transmitted to the second computing unit; If the inferred identifier represents an inferred value, the first data is input into the prediction model to obtain a candidate sequence; Alternatively, if the inferred identifier is a definite value, a decoding task is performed based on the first data to obtain the inference result.

13. The inference optimization method for the pre-trained model according to claim 1, characterized in that, After the target layer performs the second subtask based on the first data to obtain the inference result of the pre-trained model, the method further includes: Obtain operational data for each task stage; Based on the operational data, a credit value is determined, and the system's processing rate is adjusted accordingly.

14. The inference optimization method for the pre-trained model according to claim 13, characterized in that, The operational data includes the system's free memory, total memory, and fourth utilization rate. Determining the credit value based on the operational data includes: The fourth coefficient is determined based on the quotient of the free memory and the total memory; A first value is determined based on the fourth utilization rate and the utilization rate threshold; wherein the utilization rate threshold is a preset maximum acceptable utilization rate of the system; The credit value is determined based on the product of the first value and the fourth coefficient.

15. The inference optimization method for the pre-trained model according to claim 13, characterized in that, Based on the credit score, adjust the system's processing rate, including: Determine the amount of credit change based on continuous credit scores; The second value is determined based on the product of the credit change and the preset adjustment coefficient; Based on the second value and the first processing rate, a new processing rate is determined, and the system's processing rate is adjusted to the new processing rate; wherein, the first processing rate is the collected system processing rate.

16. The inference optimization method for the pre-trained model according to claim 13, characterized in that, Before adjusting the system's processing rate based on the credit value, the method further includes: If the credit change corresponding to the credit value is greater than or equal to the preset credit threshold, the system processing rate is adjusted according to the credit value. Alternatively, if the credit change corresponding to the credit value is less than a preset credit threshold, the system's processing rate is maintained; wherein, the preset credit threshold is the credit change corresponding to a significant change in resources.

17. The inference optimization method for the pre-trained model according to claim 1, characterized in that, The first data includes at least one of a first tensor, a second tensor, and a third tensor, and the step of transmitting the first data to the second computing unit through a double buffering mechanism includes: In the first cycle, the first tensor is loaded through the first buffer, the second tensor is transmitted through the second buffer, and the second subtask is executed by the second computing unit using the second tensor transmitted through the second buffer. In the second cycle, the first tensor is transmitted through the first buffer, the next batch of the third tensor is loaded through the second buffer, and the second subtask is executed by the second computing unit using the first tensor transmitted through the first buffer; wherein the first cycle and the second cycle are two consecutive cycles.

18. An electronic device, characterized in that, The electronic device includes: A memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform an inference optimization method for a pre-trained model as described in any one of claims 1 to 17.

Images