A green and ubiquitous distributed reasoning service method, apparatus, equipment and medium

Abstract

This application relates to the field of data processing technology, specifically to a green and ubiquitous distributed inference service method, apparatus, device, and medium, aiming to provide a green and ubiquitous distributed large model inference service. The method includes: grouping received inference requests into a task queue; sequentially executing pre-filling tasks using pre-filled instances; adding the decoding tasks of the model corresponding to the inference request to a working list, placing decoding tasks corresponding to the same group of inference requests in the same batch, with each batch of decoding tasks having a corresponding preset execution time; cyclically executing the decoding tasks in the working list using decoding instances; placing the key-value cache generated during task execution into a key-value cache block pool; switching the corresponding model and loading the corresponding key-value cache while executing pre-filling and decoding tasks; and obtaining multiple inference results corresponding to multiple groups of inference requests after all decoding tasks have been completed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a green and ubiquitous distributed reasoning service method, apparatus, device, and medium. Background Technology

[0002] An inference service system is a service system specifically built for large models. It provides a vast number of models and offers corresponding inference services for different user requests. In existing technologies, inference service systems include a large number of models. To ensure inference performance, the system reserves at least one GPU (Graphics Processing Unit) instance for each model in each service.

[0003] In existing technologies, a model request is inferred through a single GPU instance. For a new model request, the GPU instance must be released after the previous request has finished executing before inference can begin. When there are insufficient GPU instances, this can easily lead to excessively long inference times for the model, reducing the system's inference efficiency. Summary of the Invention

[0004] This application provides a green and ubiquitous distributed reasoning service method, apparatus, device, and medium, aiming to provide green and ubiquitous distributed large model reasoning services.

[0005] The first aspect of this application provides a green and ubiquitous distributed reasoning service method, the method comprising:

[0006] The received multiple inference requests are grouped and scheduled according to the corresponding models to obtain multiple task queues corresponding to the multiple groups of inference requests;

[0007] The pre-filled instances are executed sequentially to pre-fill the model corresponding to each group of inference requests in the task queue.

[0008] Add the decoding tasks of the models corresponding to multiple sets of inference requests to a work list. In the work list, the decoding tasks corresponding to the same set of inference requests are placed in the same batch, and the decoding tasks in each batch have a corresponding preset execution time.

[0009] The decoding instance executes the decoding tasks in the work list in a loop according to the execution time corresponding to each group of decoding tasks;

[0010] The key-value cache generated by the pre-filling task and the decoding task corresponding to each group of inference requests is stored in the key-value cache block pool of the model corresponding to the inference request;

[0011] When executing the pre-filling task and decoding task corresponding to each group of inference requests, switch the corresponding model; and when executing the decoding task, retrieve the key-value cache corresponding to the model from the key-value cache block pool.

[0012] After the pre-filling task and each step of the decoding task corresponding to each group of inference requests are completed, the single-step inference result corresponding to the inference request is obtained.

[0013] Optionally, before executing the task corresponding to each set of inference requests using pre-filled instances and decoding instances, the method further includes:

[0014] Before performing the corresponding inference task through the pre-filled instance and the decoded instance, multiple models corresponding to multiple sets of inference requests are determined.

[0015] Load the components of the inference engine into the pre-filled instance and the decoding instance;

[0016] Based on the task queue and the work list, the weight parameters of the model corresponding to the next set of inference requests are preloaded.

[0017] Optionally, before sequentially executing the pre-filled tasks corresponding to each group of inference requests through pre-filled instances, the method further includes:

[0018] The inference requests that use the same type are grouped together to obtain multiple groups of inference requests;

[0019] Each group of inference requests is placed into the corresponding task queue according to the load of the pre-filled instance, resulting in multiple groups of task queues.

[0020] Optionally, before executing the decoding tasks in the work list cyclically according to the execution time corresponding to each group of decoding tasks through the decoding instance, the method further includes:

[0021] The execution time for each group of decoding tasks in the work list is determined based on the execution time, deadline, model switching time, and service level target for each decoding task.

[0022] Optionally, the step of executing the decoding tasks in the work list cyclically according to the execution time corresponding to each group of decoding tasks through the decoding instance includes:

[0023] The decoding instance executes the first batch of decoding tasks in the work list;

[0024] If the preset execution time for executing the first batch of decoding tasks exceeds the preset execution time, the current model will be switched to the model corresponding to the next group of inference requests in the work list.

[0025] After completing one round of the decoding tasks in the work list, the decoding tasks in the work list are executed from the beginning until all the decoding tasks in the work list have been completed.

[0026] Optionally, storing the key-value cache generated by the pre-filling task and the decoding task corresponding to each group of inference requests into the key-value cache block pool of the model corresponding to the inference request includes:

[0027] Determine the address of each key-value cache block in the key-value cache block pool of the model corresponding to the inference request;

[0028] If it is determined that there is a free block in the key-value cache block pool, the key-value cache generated by the model is sent to the address corresponding to the free block.

[0029] Optionally, the step of switching the corresponding model when performing the pre-filling and decoding tasks corresponding to each group of inference requests, and loading the key-value cache corresponding to the model when performing the decoding task, includes:

[0030] While executing the pre-filling task and decoding task corresponding to each group of inference requests, the weight parameters of the model corresponding to the inference request are searched in the model pool;

[0031] If the weight parameter is found, the weight parameter is copied through the buffer to the pre-filled instance and the decoding instance;

[0032] When performing the decoding task, the key-value cache of the model corresponding to the inference request is obtained from the key-value cache block pool;

[0033] The key-value cache is loaded into the decoding instance.

[0034] A second aspect of this application provides a green and ubiquitous distributed reasoning service device, the device comprising:

[0035] The request grouping module is used to group and schedule multiple received inference requests according to the corresponding model and the load of the pre-populated instance, so as to obtain multiple task queues corresponding to the multiple groups of inference requests.

[0036] The pre-filling execution module is used to sequentially execute the pre-filling tasks of the models corresponding to each group of inference requests in the task queue through the pre-filling instances;

[0037] The work list creation module is used to add the decoding tasks of the model corresponding to multiple sets of inference requests to the work list according to the load of the decoding instance. The work list places the decoding tasks corresponding to the same set of inference requests in the same batch, and the decoding tasks in each batch have a corresponding preset execution time.

[0038] A decoding execution module is used to cyclically execute the decoding tasks in the work list according to the execution time corresponding to each group of decoding tasks through the decoding instance;

[0039] The key-value caching module is used to cache the key values ​​generated by the pre-filling task and the decoding task corresponding to each group of inference requests to the key-value cache block pool of the model corresponding to the inference request;

[0040] The model switching module is used to switch the corresponding model when executing the pre-filling task and decoding task corresponding to each group of inference requests, and to obtain the key-value cache corresponding to the model from the key-value cache block pool when executing the decoding task.

[0041] The result acquisition module is used to obtain the single-step inference result corresponding to the inference request after the pre-filling task and each step of the decoding task corresponding to each group of inference requests have been completed.

[0042] Optionally, the device further includes:

[0043] The model determination module is used to determine multiple models corresponding to multiple sets of inference requests before performing the corresponding inference task through the pre-filled instance and the decoded instance;

[0044] A component loading module is used to load components of the inference engine into the pre-filled instance and the decoding instance;

[0045] The weight parameter loading module is used to load the weight parameters of the model corresponding to the currently processed inference request into the pre-filled instance and the decoding instance;

[0046] The model prefetching module is used to preload the weight parameters of the model corresponding to the next set of inference requests based on the task queue and the work list.

[0047] Optionally, the request grouping module includes:

[0048] The request grouping submodule is used to group inference requests that use the same type into the same group to obtain multiple groups of inference requests.

[0049] The task queue establishment submodule is used to place each group of inference requests into the corresponding task queue according to the load of the pre-filled instance, thereby obtaining multiple groups of task queues.

[0050] Optionally, the device further includes:

[0051] The execution time determination module is used to determine the execution time of each group of decoding tasks in the work list based on the execution time, deadline, model switching time and service level target of each decoding task.

[0052] Optionally, the decoding execution module includes:

[0053] The first decoding execution submodule is used to execute the first batch of decoding tasks in the work list through the decoding instance;

[0054] The second decoding execution submodule is used to switch the current model to the model corresponding to the next group of inference requests in the work list if the preset time for executing the first batch of decoding tasks exceeds the preset execution time.

[0055] The third decoding execution submodule is used to continue executing the decoding tasks in the work list from the beginning after completing one round of the decoding tasks in the work list, until all the decoding tasks in the work list have been executed.

[0056] Optionally, the key-value caching module includes:

[0057] The address determination submodule is used to determine the address of each key-value cache block in the key-value cache block pool of the model corresponding to the inference request;

[0058] The key-value cache sending submodule is used to send the key-value cache generated by the model to the address corresponding to the free block when it is determined that there is a free block in the key-value cache block pool.

[0059] Optionally, the model switching module includes:

[0060] The weight parameter lookup submodule is used to look up the weight parameters of the model corresponding to the inference request in the model pool when executing the pre-filling task and decoding task corresponding to each group of inference requests.

[0061] The parameter loading submodule is used to copy the weight parameter through a buffer to the pre-filled instance and the decoding instance when the weight parameter is found.

[0062] The key-value cache retrieval submodule is used to retrieve the key-value cache of the model corresponding to the inference request from the key-value cache block pool when the decoding task is executed.

[0063] The key-value cache loading submodule is used to load the key-value cache into the decoding instance.

[0064] A third aspect of this application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the first aspect of this application.

[0065] A fourth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the first aspect of this application.

[0066] The green and ubiquitous distributed inference service method provided in this application is adopted. Multiple received inference requests are grouped and scheduled according to the load of the corresponding model and pre-filled instances, resulting in multiple task queues corresponding to multiple groups of inference requests. The pre-filled instances sequentially execute the pre-filling tasks of the models corresponding to each group of inference requests in the task queues. Decoding tasks of the models corresponding to multiple groups of inference requests are added to a working list according to the load of the decoding tasks. Decoding tasks corresponding to the same group of inference requests are placed in the same batch in the working list, and each batch of decoding tasks has a corresponding preset execution time. The decoding instances cyclically execute the decoding tasks in the working list according to the execution time corresponding to each group of decoding tasks. The key-value caches generated by the pre-filled tasks and decoding tasks corresponding to each group of inference requests are stored in the key-value cache block pool of the model corresponding to the inference request. While executing the pre-filled tasks and decoding tasks corresponding to each group of inference requests, the corresponding model is switched; and while executing decoding tasks, the key-value cache corresponding to the model is retrieved from the key-value cache block pool. After all decoding tasks corresponding to multiple groups of inference requests have been executed, multiple inference results corresponding to multiple groups of inference requests are obtained.

[0067] This method constructs a multi-language model inference system that utilizes hardware resources at a flag level. It decouples the pre-filling and decoding tasks of large language models, groups and schedules requests for the same model, and executes pre-filling tasks through pre-filling instances to avoid frequent model switching. A batch-based round-robin scheduling strategy is adopted for decoding tasks to maximize the time-sharing reuse of decoding instances while meeting the target service time. By setting a corresponding key-value cache block pool for each model, the caching and transmission of key values ​​are accelerated, ensuring that inference tasks for multiple models are completed within the target service time. Inference is completed through the collaboration of multiple nodes, and the inference system can be accessed and run through various terminals, saving server resources and achieving green and ubiquitous distributed multi-language model inference. Attached Figure Description

[0068] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a distribution chart of multi-model requests over a 24-hour period;

[0070] Figure 2 This is a schematic diagram of two resource reuse strategies with different granularities proposed in an embodiment of this application;

[0071] Figure 3 This is a schematic diagram of the inference system workflow proposed in one embodiment of this application;

[0072] Figure 4 This is a flowchart of a reasoning service method proposed in an embodiment of this application;

[0073] Figure 5 This is a schematic diagram of a reasoning service device according to an embodiment of this application;

[0074] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0075] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0076] Large Language Models (LLMS) are widely used today. Each model has different requirements in terms of deployment and performance. Inference services for large language models place higher demands on the resource efficiency of the inference system. On the one hand, each model's service needs to meet the deadline at the token level, corresponding to two metrics: Time To First Token (TTFT), which measures the latency of the large model's prefill phase, and Total Blocking Time (TBT), which measures the latency of each subsequent token generated during the large model's decoding phase. These two metrics ensure the overall service-level objective of the model service, i.e., the target time spent by the inference service. On the other hand, the system for inference services exhibits a high degree of skewness, as seen in the following reference... Figure 1 , Figure 1 It is a distribution chart of multi-model requests within 24 hours, such as Figure 1 As shown, a large number of inference requests are concentrated on a few popular models, while the remaining requests are distributed across a series of less frequently used models. Low-frequency models account for 90% of the total number of models. To ensure inference performance, at least one inference instance needs to be reserved for each model. However, this strategy increases operating expense (OPEX), requiring at least 17.7% of GPU resources to be reserved to process models that only account for 1.35% of requests, resulting in severely inefficient resource utilization.

[0077] To achieve server-insensitive computing, existing research proposes multiplexing a single GPU instance to serve multiple models. This includes accelerating model weight loading to reduce overhead during model switching, scheduling model requests to optimize the waiting time of each request, and maximizing the possibility of multiplexing through model parallelism strategies and placement schemes. These methods improve resource utilization by reserving instances only for active models (i.e., models with at least one request being served). However, for models used less frequently in real-world loads, the long and unpredictable output length of large language model requests leads to a large number of models remaining active even with low request arrival efficiency. For example, even with a total request arrival rate of only 3.7 requests per second, a large number of models remain active. The drawback is that these systems perform time-sharing multiplexing at the request granularity. In such systems, the execution of a new model request must wait for the completion of previous requests to release GPU (Graphics Processing Unit) instances. If the number of reserved GPU instances is less than the number of active models, it will lead to severe head-of-line blocking, resulting in a breach of the service level target for waiting models, i.e., a breach caused by the failure to complete inference service within the second service time.

[0078] This application develops a solution for flag granularity to overcome the inherent limitations of existing methods, referencing Figure 2 , Figure 2 This diagram illustrates two resource reuse strategies at different granularities proposed in an embodiment of this application. The upper part represents resource reuse strategies in related technologies, as shown in the figure. These related technologies are prone to causing header congestion. The lower part represents the resource reuse strategy proposed in this application, where P represents a pre-filling task and D represents a decoding task. In the request-granularity resource reuse strategy, the execution of a new model request must wait for the completion of a previous request to release the GPU instance. In the flag-granularity resource reuse strategy proposed in this application, flag-granularity multiplexing allows waiting requests to preempt running requests after each flag is generated, thereby alleviating header congestion and significantly reducing service level target defaults, i.e., exceeding the service target time.

[0079] refer to Figure 3 , Figure 3 This is a schematic diagram of the inference system workflow proposed in one embodiment of this application, as shown below. Figure 3As shown, the inference service system mainly consists of three components: a proxy layer, a GPU pool (prefill instances and decode instances), and a memory management system. The proxy layer is responsible for receiving and scheduling inference requests, ensuring load balancing and guaranteeing system fault tolerance. State synchronization between the proxy layer and other components is achieved through a shared memory mechanism, such as using Redis (a key-value database). The system constructed in this application can schedule requests from different models to the same instance, such as... Figure 3 As shown in ①, ②, and ⑥ in the figure, the GPU pool, as a resource pool, contains a group of GPU instances virtualized by a cloud service provider. Each instance may contain one or more GPUs hosted on physical machines. In this application, a single instance performs either a pre-filling or decoding task (corresponding to ③, ⑤, and ⑧ in the figure) and can serve requests from different models according to the guidance of the flag-level scheduler. Model switching is a critical operation, as shown in ④, ⑦, and ⑨ in the figure. The Scheduler is the agent program. The memory manager coordinates the CPU (Central Processing Unit) memory and idle GPU memory of each node in the service cluster, mainly serving two subsystems: the first is the Quickloader (fast startup component), which uses available memory to cache model weights, accelerating the time to pull model weights from a remote model repository. The second is unified CPU swapping, which uses CPU memory to store swapped-out key-value caches.

[0080] refer to Figure 4 , Figure 4 This is a flowchart of a reasoning service method proposed in an embodiment of this application. Figure 4 As shown, the method includes the following steps:

[0081] S11: The received multiple inference requests are grouped and scheduled according to the load of the corresponding model and pre-filled instance to obtain multiple task queues corresponding to the multiple groups of inference requests, and each task queue corresponds to a pre-filled instance.

[0082] In this embodiment, an inference request is a question or request submitted by a user to the inference service system that requires inference, such as answering a question or generating an image. The task queue contains pre-filled tasks corresponding to the inference requests, and pre-filled tasks corresponding to inference requests for the same model are in the same task queue.

[0083] In this embodiment, the inference service system maintains a task queue on each pre-filled instance. Each task queue contains pre-filled tasks corresponding to the same group of requests. When multiple inference requests are received, the model corresponding to each inference request is first determined. Then, inference requests for the same model are grouped together, and each group of inference requests is added to the corresponding task queue. Each task queue corresponds to one pre-filled instance.

[0084] In this embodiment, refer to the following code:

[0085] Algorithm 1 Grouped Prefill-Phase Scheduling

[0086] Input: Prefill instances IP.

[0087] 1:>Event: On arrival of request r:

[0088] 2:min_load←oo

[0089] 3: i*← Ip[0]

[0090] 4: for all instances i in IP do

[0091] 5: for all group g in i.job_queue do

[0092] 6:if g.model = r.model and g.size <MAX_GPSIZE then

[0093] 7: Add r to group g # Prioritize existing groups

[0094] 8: return

[0095] 9: load ← Total time to execute all groups in i

[0096] 10: if load <min_load then

[0097] 11: min_load ← load # Pick the least loaded instance

[0098] 12: i*←i

[0099] 13:Make a group with r and append to i*.job_queue

[0100] 14:>Event: On selecting a batch for prefll instance i execution:

[0101] 15:Select one request from the front group of i.job_queue

[0102] The code above is the core of the group scheduling strategy. Its key idea is to group requests for the same model to minimize the number of model switches. Each pre-filled instance maintains a task queue containing grouped tasks. When a new pre-filled task is received, the algorithm first adds the task to an existing group. If no suitable group is found, it means the model corresponding to the request is different from the model corresponding to the previous group—it's a new model. In this case, a new group is created and added to the task queue with the lowest load. The load of the task queue is defined as the total time required to complete all pending groups, including task execution time and switching time. The batch size of the pre-filled instance can be limited to 1 (i.e., only one pre-filled task is executed from the front of the task queue at a time). This is because the execution time of a pre-filled batch increases approximately linearly with the number of pre-filled flags; smaller batches can reduce overall waiting time without significantly affecting throughput.

[0103] In this embodiment, the GPU pool is divided into two partitions, one dedicated to pre-filling and the other to decoding. When a request is received, the pre-filling task is first scheduled using instances in the pre-filling partition, and then the subsequent decoding task is scheduled using instances in the decoding partition. The first-come-first-served (FCFS) scheduling strategy is used to avoid frequent model switching and minimize the first flag time.

[0104] In this embodiment, the specific steps of grouping and scheduling multiple received inference requests according to their corresponding models to obtain multiple task queues corresponding to multiple groups of inference requests include:

[0105] S11-1: Group the inference requests that use the same type into the same group to obtain multiple groups of inference requests.

[0106] In this embodiment, when multiple model inference requests are received, inference requests using the same model are treated as the same set of inference requests, thereby obtaining multiple sets of inference requests.

[0107] For example, if 10 inference requests are received, where the first 5 inference requests require the use of model A and the sixth 10 inference requests require the use of model B, then the first 5 inference requests are treated as the first group of inference requests, and the sixth 10 inference requests are treated as the second group of inference requests.

[0108] S11-2: Place each group of inference requests into the corresponding task queue according to the load of the pre-filled instance, to obtain multiple groups of task queues.

[0109] In this embodiment, after the received inference requests are grouped, each group of inference requests is placed into the corresponding task queue. When placing the requests, the load of the task queue corresponding to the pre-filled instance is determined, and newly received inference requests are preferentially placed into the task queue with a smaller load.

[0110] S12: Execute the pre-filling tasks of the models corresponding to each group of inference requests in the task queue in sequence through the pre-filled instances.

[0111] In this embodiment, a pre-filled instance is an instance composed of multiple GPUs used to perform pre-filling tasks for large models. Pre-filling refers to preprocessing the input sequence and calculating key intermediate states (such as the Key-Value Cache in the attention mechanism) before model inference, providing cached data for the subsequent decoding stage to reduce redundant computation. For example, in a dialogue scenario, pre-filling encodes the complete prompt input by the user in advance to generate reusable intermediate tensors.

[0112] In this embodiment, the pre-filling instance first executes the pre-filling task for the model corresponding to each group of inference requests in the task queue sequentially. In the GPU pool, the pre-filling instance and the decoding instance are composed of different GPUs, with the pre-filling instance specifically responsible for executing the pre-filling task. When executing the pre-filling task, the pre-filling instance reads the first inference request added in a group of inference requests, processes its input, and obtains the corresponding key-value pair. The pre-filling instance has already loaded the corresponding model's weight parameters when executing the pre-filling task.

[0113] S13: The decoding tasks of the models corresponding to the multiple sets of inference requests are added to multiple work lists according to the load of the decoding instances. Each work list corresponds to a decoding instance. The decoding tasks corresponding to the same set of inference requests are placed in the same batch in the work list. The decoding tasks in each batch have a corresponding preset execution time.

[0114] In this embodiment, the decoding task generates subsequent sequences step by step based on the key values ​​output from the pre-filled sequence. The working list is a list set in the decoding instance, which includes multiple batches of decoding tasks, and the preset execution time is the window time for executing each batch of decoding tasks.

[0115] In this embodiment, each decoding instance maintains a rotating work list, meaning each decoding instance corresponds to one work list. Each batch in the list consists of requests for the same model, and each batch is allocated a fixed time quota. Upon receiving multiple sets of inference requests, the decoding tasks for the model corresponding to each set of inference requests are added to the work list. Decoding tasks corresponding to the same set of inference requests are placed in the same batch. Each batch of decoding tasks has a corresponding preset execution time. Furthermore, based on the load of the decoding instances, tasks are preferentially placed in decoding instances with lower loads.

[0116] For example, in the work list, the preset execution time for the first inference request is 2 seconds, and the preset time for the second inference request is 3 seconds.

[0117] S14: The decoding tasks in the work list are executed cyclically according to the execution time corresponding to each group of decoding tasks by the decoding instance.

[0118] In this embodiment, at the beginning of each round of inference, the execution time corresponding to each batch of decoding tasks is calculated, and then each batch of decoding tasks is executed according to the execution time. After the execution time of a batch of decoding tasks exceeds the preset execution time, the model is quickly switched and the next batch of decoding tasks is executed. This process is repeated to execute multiple batches of decoding tasks in the list until all decoding tasks are completed.

[0119] In this embodiment, the step of executing the decoding tasks in the work list cyclically according to the execution time corresponding to each group of decoding tasks by the decoding instance includes:

[0120] S14-1: Execute the first batch of decoding tasks in the work list through the decoding instance.

[0121] In this embodiment, when the decoding instance executes each round of decoding tasks, it first executes the first batch of decoding tasks in the work list, that is, it generates subsequent key values ​​by using the key values ​​obtained from the pre-filling tasks corresponding to the first batch of inference requests.

[0122] S14-2: If the preset execution time for executing the first batch of decoding tasks exceeds the preset execution time, the current model is switched to the model corresponding to the next group of inference requests in the work list.

[0123] In this embodiment, if the preset execution time for the first batch of decoding tasks exceeds the preset execution time, the current model is switched to the model corresponding to the next set of inference requests in the cache. Here, the GPU will pre-obtain the model corresponding to the next batch of decoding tasks.

[0124] S14-3: After completing one round of the decoding tasks in the work list, continue to execute the decoding tasks in the work list from the beginning until all the decoding tasks in the work list have been completed.

[0125] In this embodiment, after completing one round of decoding tasks in the work list, the decoding tasks in the work list are executed from the beginning until all decoding tasks in the work list are completed. Before each round of decoding tasks is executed, the preset execution time of each batch of decoding tasks in the work list is calculated to ensure that the list is updated every round.

[0126] In this embodiment, the scheduling design for the decoding task is based on the uniqueness of large language model inference. The execution of the decoding task is multi-round, and the output flags can be buffered. Let t be the time required for one decoding step, and d be the deadline for that step (i.e., the time between target flags, TBT). For every consecutive n decoding steps, the request can tolerate a maximum latency of n(dt) time. As long as the latency meets this time, it does not violate the service level objective. Since t is usually small (e.g., tens of milliseconds), and d is relatively generous (e.g., for chat applications, TBT is 100 milliseconds), this extra leeway can be used to serve other requests.

[0127] Please refer to the following code:

[0128] Algorithm 2 Batched Decoding-Phase Scheduling

[0129] Input: Decoding instance IP.

[0130] 1:>Event: On arrival of request r:

[0131] 2:Dispatch as in Algorithm 1, deriving max batch sizes from the KVcache capacity on GPU, using work list sizes for load

[0132] 3:>Event: Always on decoding instance i:

[0133] 4: while True do

[0134] 5: # Start of a round

[0135] 6: Reorder i.work_list to group batches with the same model

[0136] 7: Assign time quota q_k to the k-th batch in i.work_list

[0137] 8: n_turn ← Number of batches in i.work_list

[0138] 9: for all index k in 1 ~ n_turn do

[0139] 10: # Start of a turn

[0140] 11: Decode the k-th batch in i.work_list for q_k seconds

[0141] This code is the core code of the round-robin scheduling scheme. The idea is that each decoding instance maintains a rotating work list. Each batch of decoding tasks in the list consists of decoding tasks corresponding to requests for the same model and is allocated a time quota. Newly pre-filled completed requests are scheduled through a grouping mechanism, but the load is calculated based on the size of the work list, and the batch size limit is determined by the key-value cache capacity on the GPU. The execution of the work list is organized in rounds.

[0142] Before executing the decoding tasks in the work list cyclically according to the execution time corresponding to each group of decoding tasks through decoding instances, the system determines the execution time corresponding to each group of decoding tasks in the work list based on the execution time, deadline, model switch time, and service level objective of each decoding task. The algorithm allocates a time quota qi to the i-th batch using the following formula:

[0143] (1)

[0144] in c is the sum of the switching costs of all unique models in the work list, and α is given by the following formula:

[0145] (2)

[0146] α is the reciprocal of the service level target achievement for each round. The algorithm reorders the work list after each round of decoding tasks to ensure that batches sharing the same model are placed adjacently. This can happen if a batch requires more key-value cache space than is available on the GPU. Each batch decodes for qi time, and this formula ensures... This means that the quota allocated to the i-th batch allows for a latency window time equal to 1 / α of the total round time. In this way, all batches of decoding tasks in each round will always achieve a guaranteed service level target, ensuring that the decoding task corresponding to each inference request obtains the decoding result within the target service time, thus providing the inference result to the user.

[0147] S15: Store the key-value cache generated by the pre-filling task and the decoding task corresponding to each group of inference requests into the key-value cache block pool of the model corresponding to the inference request.

[0148] In this embodiment, the key-value cache is a core technology for optimizing the inference efficiency of large language models. In generative tasks (such as text generation), the self-attention weights of the historical sequence need to be repeatedly calculated each time the decoding is performed. The key-value cache avoids the repeated calculation of the historical sequence by caching the generated keys and values. The key-value cache block pool is a cache block pool built based on the slab allocation mechanism of memory blocks. Each key-value cache block pool corresponds to a model, and the key-value shapes of each model may be different or the same.

[0149] In this embodiment, when performing pre-filling and decoding tasks for each set of inference requests, a key-value cache is generated. At this time, the key-value cache generated by the pre-filling and decoding tasks is stored in the key-value cache block pool of the model corresponding to the inference request.

[0150] In this embodiment, a slab allocation algorithm is used to achieve unified storage in CPU memory for storing key-value caches, balancing the memory waste caused by internal fragmentation during memory allocation and the time overhead of fine-grained memory management. For cache blocks of different shapes from different models, a unified CPU cache is managed through the slab allocation mechanism. The cache space is divided into fixed-size blocks called slabs, and each slab is assigned to a specific key-value cache shape, serving as a pool of key-value cache blocks for that shape. When allocating memory, the system first uses free blocks from slabs of the same shape. If no free blocks are available, the system acquires a new slab for that shape. The system tracks each block by calculating a global offset from the starting position of the same CPU cache and only considers the existence of slabs when allocating blocks.

[0151] In this embodiment, a balance is achieved between management overhead and internal fragmentation through the slab allocation mechanism. The degree of fragmentation is limited to the size of the slab, and slab allocation achieves efficient memory utilization under real workloads.

[0152] In this embodiment, storing the key-value cache generated by the pre-filling task and the decoding task corresponding to each group of inference requests into the key-value cache block pool of the model corresponding to the inference request includes:

[0153] S15-1: Determine the address of each key-value cache block in the key-value cache block pool of the model corresponding to the inference request.

[0154] In this embodiment, when a key-value cache is generated for the model corresponding to the inference request, the address of the key-value cache block pool of the model corresponding to the inference request in memory is determined. The location of the key-value cache block pool is a region in memory, which has a corresponding start address and end address.

[0155] S15-2: If it is determined that there is a free block in the key-value cache block pool, the key-value cache generated by the model is sent to the address corresponding to the free block.

[0156] In this embodiment, after determining the address of the key-value cache block pool corresponding to the model, it is determined whether there are free blocks in the key-value cache block pool. If it is determined that there are free blocks in the key-value cache block pool, the key-value cache generated by the model is sent to the corresponding key-value cache block pool.

[0157] S16: When executing the pre-filling task and decoding task corresponding to each group of inference requests, switch the corresponding model, and when executing the decoding task, retrieve the key-value cache corresponding to the model from the key-value cache block pool.

[0158] In this embodiment, when the inference service system executes the pre-filling task and decoding task corresponding to each group of inference requests, it needs to switch the model in the pre-filling instance and decoding instance to the corresponding model, and obtain the key value that the model has stored from the key-value cache block pool, and then execute the inference task.

[0159] In this embodiment, a high-speed model switching technique is used during model switching. This technique is based on the analysis of real-world inference system operation. Switching models on an instance is limited not only by the latency of loading model weights from CPU memory to the GPU, but also by the restart and initialization overhead of other components of the inference system. The following factors significantly affect model switching time: Distributed executors: Inference engines support model parallelism through distributed executors (such as Ray and NCCL), whose initialization typically takes tens of seconds. Performance analysis and optimization: Some engines perform performance analysis and optimization (e.g., determining key-value cache size or using CUDA graphs for computational acceleration), which also takes tens of seconds. Model weight loading: Loading model weights onto the GPU is a time-consuming process, and the latency increases with the model size. Although modern PCIe buses offer high theoretical bandwidth (e.g., PCIe 4.0 has a bandwidth of 32GB / s), inference engines often fail to fully utilize this bandwidth. For example, loading the LLaMA-13B model via PCIe 4.0 (using 2x tensor parallelism) takes approximately 4.6 seconds in micro-benchmarks, corresponding to a practical bandwidth of only 2.83 GB / s. Key-value cache initialization: Inference engines that use CPU memory to store key-value caches lock memory pages to improve data access performance. However, locking memory pages introduces a second-level overhead. Other components: The initialization of other components (such as the scheduler) also impacts time overhead.

[0160] To address the aforementioned issues, the inference service system employs a pre-caching approach to avoid redundant initialization of components outside the model, and introduces model weight caching, pipelined loading, and model prefetching to accelerate the loading of model weights.

[0161] In this embodiment, for the key-value cache transfer, block movement synchronization is only forced in the following four situations. These situations involve inherent data dependencies: model inference requires waiting for the block to be loaded onto the GPU; model switching requires waiting for the block to be swapped out into CPU memory; block movement requires waiting for the source data to be ready; and block movement requires waiting for the target block to be cleared. Synchronization operation refers to confirming the completion of data transmission after the data transfer. Apart from these four situations, no synchronization operation is required, thus ensuring that the key-value cache transfer does not incur excessive latency.

[0162] In this embodiment, CUDA Event (Compute Unified Device Architecture, Event) is used to efficiently synchronize block movement. CUDA Event is a core tool for implementing asynchronous GPU task management and performance tuning. On each instance of the GPU, the system maintains an independent CUDA stream. Each block movement is captured as a new Event (event object) by cudaEventRecord (used to insert an event into the specified CUDA stream to record the current GPU execution time). For the first and second scenarios, the instance pauses inference or model switching, using `cudaEventSynchronize` (used to force the host thread to wait for a specified CUDA event to complete recording, ensuring that associated GPU operations (such as kernel functions and data transfers) have finished executing) until the relevant block movement is complete. For the third scenario, the instance continues to call `cudaStreamWaitEvent` (a key API in CUDA for implementing inter-stream synchronization) to block only the relevant stream, allowing other CPU and GPU streams to continue execution. To ensure synchronization for the fourth scenario, the moving blocks and their corresponding Events are collected into a separate list, and a background thread periodically polls the `cudaEventQuery` (CUDA API for non-blocking querying of event status) list to reclaim moved blocks back to the free list. Explicit blocking is only used if the background thread fails to reclaim enough moved blocks before allocation. This design effectively offloads the synchronization task for the fourth scenario from the background thread, reducing overall overhead and consequently reducing the overhead of key-value cache transfers.

[0163] In this embodiment, when executing the pre-filling task and decoding task corresponding to each group of inference requests, the specific steps of switching the corresponding model, and when executing the decoding task, obtaining the key-value cache corresponding to the model from the key-value cache block pool include:

[0164] S16-1: When executing the pre-filling task and decoding task corresponding to each group of inference requests, look up the weight parameters of the model corresponding to the inference request in the model pool.

[0165] In this embodiment, the model pool is a region in shared memory used to store weight parameters.

[0166] In this embodiment, when executing the pre-filled task decoding task corresponding to each group of inference requests, the model corresponding to the group of inference requests is determined, and then the weight parameters of the model corresponding to the inference request are searched in the model pool.

[0167] S16-2: If the weight parameter is found, the weight parameter is copied through the buffer to the pre-filled instance and the decoding instance.

[0168] In this embodiment, the buffer is a memory area unique to each GPU.

[0169] In this embodiment, when the service system finds the weight parameters in the model pool, it copies the weight parameters through a buffer to the pre-filled instance and the decoding instance, that is, to the GPU that processes the inference request. This completes the loading of the weight parameters.

[0170] S16-3: When performing the decoding task, retrieve the key-value cache of the model corresponding to the inference request from the key-value cache block pool.

[0171] In this embodiment, when processing inference requests, key-value caches are generated and stored in a key-value cache block pool. During the cyclic execution of decoding tasks, a decoding task corresponding to an inference request may not be completed within a window period. At this time, it is necessary to store the key-value cache and wait for the next iteration of the decoding task to continue execution. Each time a model is switched, in order to continue executing the decoding task of that model, it is necessary to retrieve the key-value cache stored when the decoding task corresponding to the previous request was executed from the corresponding key-value cache block pool so as to continue the previous decoding task.

[0172] S16-4: Load the key-value cache into the decoding instance.

[0173] In this embodiment, after obtaining the key-value cache generated by the model corresponding to the inference request during the previous decoding task, the key-value cache is loaded into the decoding instance before the current decoding task begins. The decoding instance runs the model corresponding to the inference request, and the model continues to perform inference based on the obtained key-value cache.

[0174] S17: After the pre-filling task and each step decoding task corresponding to each group of inference requests have been completed, the single-step inference result corresponding to the inference request is obtained.

[0175] In this embodiment, after the pre-filling tasks corresponding to multiple inference requests and each step of the decoding task are completed, multiple single-step inference results corresponding to multiple inference requests are obtained. That is, after each decoding task is completed, the inference result corresponding to the inference request will be obtained.

[0176] For example, if the inference request is to generate a specific image, the inference result is the generated image; if the inference request is to generate a piece of text, the inference result is the generated text.

[0177] In this embodiment, a first-come-first-served scheduling strategy is designed for the pre-filling stage of the large language model, and a round-robin scheduling strategy is designed for the decoding stage. This ensures that inference tasks do not violate service level objectives, enabling batch processing of inference requests. Each GPU can achieve server-insensitive model switching, thereby saving server resources. Through efficient key-value cache swapping technology, a unified CPU key-value cache storage mechanism and an asynchronous key-value cache transmission mechanism are designed to optimize the memory usage of key-value cache storage and the latency overhead of transmission, thereby achieving efficient server-insensitive model service and improving system resource utilization.

[0178] In another embodiment of this application, before sequentially executing the pre-filled tasks corresponding to each group of inference requests through pre-filled instances, the method further includes:

[0179] S21: Before performing the corresponding inference task through the pre-filled instance and the decoded instance, determine multiple models corresponding to multiple sets of inference requests.

[0180] In this embodiment, different inference requests are processed using different models. When the inference service system receives an inference request from a user, it parses the request to obtain the content to be inferred and the model required. Each set of inference requests corresponds to one model.

[0181] S22: Load the components of the inference engine into the pre-filled instance and the decoding instance.

[0182] In this embodiment, the components of the large model are some basic components that make up the large model, and different large models have the same components.

[0183] In this embodiment, the service system initializes the inference engine only once on each instance and caches components other than the model and key-value cache. When initializing the inference engine, the components corresponding to the inference engine are loaded into the pre-filled instance and the decoding instance.

[0184] For example, the components are general components such as decoders, encoders, attention mechanisms, etc.

[0185] In this embodiment, by pre-caching the components of the model in the instance, when switching models, it is only necessary to obtain the parameters such as the weight of each model. In this way, the loading of general components will not take up the model switching time, thus accelerating the model switching.

[0186] Furthermore, the service system performs performance analysis on each model in advance, such as analyzing the size of the model's key-value cache. This analysis determines the switching and loading time for each model. The unified CPU cache also eliminates the need to lock memory pages during model switching.

[0187] S23: Load the weight parameters of the model corresponding to the inference request being processed into the pre-filled instance and the decoding instance.

[0188] In this embodiment, before executing the inference task corresponding to the inference request, the weight parameters of the model corresponding to the current inference request are loaded into the pre-filled instance and the decoding instance to complete the model loading.

[0189] In this embodiment, a quickloader (dynamic loading component) is introduced. This component consists of a model pool in shared memory and a locked buffer unique to each GPU. The model cache pool comes from the original tensor blocks in the checkpoint file. When the service system issues a model switching request, the quickloader first checks the required weights in the model pool. If the corresponding weights are not found in the model pool, they are pulled from a remote model repository. After the weights are obtained, the locked buffer is copied to the GPU. This copying process is multi-threaded, block-based, and pipelined, thus making full use of PCIe bandwidth and asynchronous data transmission, thereby accelerating model switching. For models cached in the model pool, the built system can achieve model loading speeds in the order of seconds.

[0190] S24: Based on the task queue and the work list, preload the weight parameters of the model corresponding to the next set of inference requests.

[0191] In this embodiment, the task queue stores the pre-filled tasks corresponding to each group of inference requests, and the work list stores the decoding tasks corresponding to each group of inference requests. These tasks are arranged in order. Based on the task queue and the work list, the model corresponding to the next group of inference requests can be determined. After determining the model corresponding to the next group of inference requests, the corresponding weight parameters are obtained from the model pool, and these weight parameters are preloaded to reduce the latency of model switching.

[0192] In this embodiment, in order to further reduce the overhead of model switching, the next required model is prefetched based on the pre-filled task queue and the decoding work list, and the data transmission and GPU computation are overlapped. This method is particularly effective for decoding instances because the time quota for each round usually completely masks the time overhead of switching, thereby achieving instant switching without the server's awareness.

[0193] In the embodiments described above, this application proposes a service level target-aware flag-level request scheduling technology, a high-speed model switching technology, and an efficient key-value cache swapping-in / swap-out technology. On one hand, a first-come-first-served algorithm and a round-robin scheduling algorithm are designed for the pre-filling and decoding tasks of large language models, respectively. Inference requests for the same model are grouped together to execute pre-filling tasks, and the decoding tasks corresponding to each group of inference requests are executed in batches, ensuring that multiple inference requests are processed simultaneously without violating service level targets, thus improving the overall efficiency of the system. High-speed model switching is achieved through caching components, fast model loading, and model prefetching. Through a unified CPU key-value cache storage mechanism and an asynchronous key-value cache transmission mechanism, the memory usage of key-value cache storage and the overhead and latency of transmission are optimized, providing efficient, server-insensitive model services, thereby saving server resources. This approach is applicable to various platforms and terminals, realizing a green and ubiquitous distributed inference service.

[0194] Based on the same inventive concept, one embodiment of this application provides a green and ubiquitous distributed inference service device. (Reference) Figure 5 , Figure 5 This is a schematic diagram of a reasoning service device 500 according to an embodiment of this application. Figure 5 As shown, the device includes:

[0195] The request grouping module 501 is used to group and schedule multiple received inference requests according to the corresponding models to obtain multiple task queues corresponding to the multiple groups of inference requests.

[0196] The pre-filling execution module 502 is used to sequentially execute the pre-filling tasks of the models corresponding to each group of inference requests in the task queue through the pre-filling instances;

[0197] The work list creation module 503 is used to add the decoding tasks of the model corresponding to multiple sets of inference requests to the work list. The work list places the decoding tasks corresponding to the same set of inference requests in the same batch, and the decoding tasks in each batch have a corresponding preset execution time.

[0198] The decoding execution module 504 is used to cyclically execute the decoding tasks in the work list according to the execution time corresponding to each group of decoding tasks through the decoding instance;

[0199] The key-value caching module 505 is used to store the key-value cache generated by the pre-filling task and the decoding task corresponding to each group of inference requests into the key-value cache block pool of the model corresponding to the inference request;

[0200] The model switching module 506 is used to switch the corresponding model when executing the pre-filling task and the decoding task corresponding to each group of inference requests, and to obtain the key-value cache corresponding to the model from the key-value cache block pool when executing the decoding task.

[0201] The result acquisition module 507 is used to obtain multiple inference results corresponding to multiple inference requests after all the decoding tasks corresponding to the multiple sets of inference requests have been completed.

[0202] Optionally, the device further includes:

[0203] The model determination module is used to determine multiple models corresponding to multiple sets of inference requests before performing the corresponding inference task through the pre-filled instance and the decoded instance;

[0204] A component loading module is used to load components of the inference engine into the pre-filled instance and the decoding instance;

[0205] The weight parameter loading module is used to load the weight parameters of the model corresponding to the currently processed inference request into the pre-filled instance and the decoding instance;

[0206] The model caching module is used to preload the weight parameters of the model corresponding to the next set of inference requests based on the task queue and the work list.

[0207] Optionally, the request grouping module includes:

[0208] The request grouping submodule is used to group inference requests that use the same type into the same group to obtain multiple groups of inference requests.

[0209] The task queue establishment submodule is used to place each group of inference requests into the corresponding task queue according to the load of the pre-filled instance, thereby obtaining multiple groups of task queues.

[0210] Optionally, the device further includes:

[0211] The execution time determination module is used to determine the execution time of each group of decoding tasks in the work list based on the execution time, deadline, model switching time and service level target of each decoding task.

[0212] Optionally, the decoding execution module includes:

[0213] The first decoding execution submodule is used to execute the first batch of decoding tasks in the work list through the decoding instance;

[0214] The second decoding execution submodule is used to switch the current model to the model corresponding to the next group of inference requests in the work list if the preset time for executing the first batch of decoding tasks exceeds the preset execution time.

[0215] The third decoding execution submodule is used to continue executing the decoding tasks in the work list from the beginning after completing one round of the decoding tasks in the work list, until all the decoding tasks in the work list have been executed.

[0216] Optionally, the key-value caching module includes:

[0217] The address determination submodule is used to determine the address of each key-value cache block in the key-value cache block pool of the model corresponding to the inference request;

[0218] The key-value sending submodule is used to send the key-value cache generated by the model to the address corresponding to the free block when it is determined that there is a free block in the key-value cache block pool.

[0219] Optionally, the model switching module includes:

[0220] The weight parameter lookup submodule is used to look up the weight parameters of the model corresponding to the inference request in the model pool when executing the pre-filling task and decoding task corresponding to each group of inference requests.

[0221] The parameter loading submodule is used to copy the weight parameter through a buffer to the pre-filled instance and the decoding instance when the weight parameter is found.

[0222] The key-value retrieval submodule is used to retrieve the key-value cache of the model corresponding to the inference request from the key-value cache block pool when the decoding task is executed.

[0223] The key-value loading submodule is used to load the key-value cache into the decoding instance.

[0224] Based on the same inventive concept, another embodiment of this application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the inference service method as described in any of the above embodiments of this application.

[0225] Based on the same inventive concept, another embodiment of this application provides an electronic device. Figure 6 This is a schematic diagram of an electronic device 600 according to an embodiment of this application, including a memory 602, a processor 601, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps in the inference service method described in any of the above embodiments of this application.

[0226] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0227] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0228] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0229] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0230] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0231] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0232] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0233] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0234] The above provides a detailed description of the green and ubiquitous distributed reasoning service method, apparatus, device, and medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A green and ubiquitous distributed reasoning service method, characterized in that, The method includes: The received multiple inference requests are grouped and scheduled according to the corresponding model and the load of the pre-filled instance to obtain multiple task queues corresponding to the multiple groups of inference requests, and each task queue corresponds to a pre-filled instance. The pre-filled instances are executed sequentially to pre-fill the model corresponding to each group of inference requests in the task queue. The decoding tasks of the models corresponding to the multiple sets of inference requests are added to multiple work lists according to the load of the decoding instance. Each work list corresponds to a decoding instance. The decoding tasks corresponding to the same set of inference requests are placed in the same batch in the work list. The decoding tasks in each batch have a corresponding preset execution time. The decoding instance executes the decoding tasks in the work list in a loop according to the execution time corresponding to each group of decoding tasks; The key-value cache generated by the pre-filling task and the decoding task corresponding to each group of inference requests is stored in the key-value cache block pool of the model corresponding to the inference request; When executing the pre-filling task and decoding task corresponding to each group of inference requests, switch the corresponding model; and when executing the decoding task, retrieve the key-value cache corresponding to the model from the key-value cache block pool. After the pre-filling task and each step of the decoding task corresponding to each group of inference requests are completed, the single-step inference result corresponding to the inference request is obtained.

2. The green and ubiquitous distributed reasoning service method according to claim 1, characterized in that, The method further includes: Before performing the corresponding inference task through the pre-filled instance and the decoded instance, multiple models corresponding to multiple sets of inference requests are determined. Load the components of the inference engine into the pre-filled instance and the decoding instance; The weight parameters of the model corresponding to the inference request being processed are loaded into the pre-filled instance and the decoding instance; Based on the task queue and the work list, the weight parameters of the model corresponding to the next set of inference requests are preloaded.

3. The green and ubiquitous distributed reasoning service method according to claim 1, characterized in that, The process of grouping and scheduling multiple received inference requests according to the load of the corresponding model and pre-filled instances results in multiple task queues corresponding to multiple groups of inference requests, including: The inference requests that use the same type are grouped together to obtain multiple groups of inference requests; Each group of inference requests is placed into the corresponding task queue according to the load of the pre-filled instance, resulting in multiple groups of task queues.

4. The green and ubiquitous distributed reasoning service method according to claim 1, characterized in that, Before executing the decoding tasks in the work list cyclically according to the execution time corresponding to each group of decoding tasks through the decoding instance, the method further includes: The execution time for each group of decoding tasks in the work list is determined based on the execution time, deadline, model switching time, and service level target for each decoding task.

5. The green and ubiquitous distributed reasoning service method according to claim 1, characterized in that, The step of executing the decoding tasks in the work list cyclically according to the execution time corresponding to each group of decoding tasks by the decoding instance includes: The decoding instance executes the first batch of decoding tasks in the work list; If the preset execution time for executing the first batch of decoding tasks exceeds the preset execution time, the current model will be switched to the model corresponding to the next group of inference requests in the work list. After completing one round of the decoding tasks in the work list, the decoding tasks in the work list are executed from the beginning until all the decoding tasks in the work list have been completed.

6. The green and ubiquitous distributed reasoning service method according to claim 1, characterized in that, The step of storing the key-value cache generated by the pre-filling task and the decoding task corresponding to each group of inference requests into the key-value cache block pool of the model corresponding to the inference request includes: Determine the address of each key-value cache block in the key-value cache block pool of the model corresponding to the inference request; If it is determined that there is a free block in the key-value cache block pool, the key-value cache generated by the model is sent to the address corresponding to the free block.

7. The green and ubiquitous distributed reasoning service method according to claim 1, characterized in that, The step of switching the corresponding model when executing the pre-filling task and decoding task corresponding to each group of inference requests, and obtaining the key-value cache corresponding to the model from the key-value cache block pool when executing the decoding task, includes: When executing the pre-filling task and decoding task corresponding to each group of inference requests, the weight parameters of the model corresponding to the inference request are searched in the model pool; If the weight parameter is found, the weight parameter is copied through the buffer to the pre-filled instance and the decoding instance; When performing the decoding task, the key-value cache of the model corresponding to the inference request is obtained from the key-value cache block pool; The key-value cache is loaded into the decoding instance.

8. A green and ubiquitous distributed inference service device, characterized in that, The device includes: The request grouping module is used to group and schedule multiple received inference requests according to the corresponding model and the load of the pre-filled instance, to obtain multiple task queues corresponding to multiple groups of inference requests, and each task queue corresponds to a pre-filled instance. The pre-filling execution module is used to sequentially execute the pre-filling tasks of the models corresponding to each group of inference requests in the task queue through the pre-filling instances; The work list creation module adds the decoding tasks of the model corresponding to multiple sets of inference requests to multiple work lists according to the load of the decoding instance. Each work list corresponds to a decoding instance. The decoding tasks corresponding to the same set of inference requests are placed in the same batch in the work list. The decoding tasks in each batch have a corresponding preset execution time. A decoding execution module is used to cyclically execute the decoding tasks in the work list according to the execution time corresponding to each group of decoding tasks through the decoding instance; The key-value caching module is used to cache the key values ​​generated by the pre-filling task and the decoding task corresponding to each group of inference requests to the key-value cache block pool of the model corresponding to the inference request; The model switching module is used to switch the corresponding model when executing the pre-filling task and the decoding task corresponding to each group of inference requests, and to obtain the key-value cache corresponding to the model from the key-value cache block pool when executing the decoding task. The result acquisition module is used to obtain the single-step inference result corresponding to the inference request after the pre-filling task and each step of the decoding task corresponding to each group of inference requests have been completed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

Images