Green ubiquitous distributed reasoning service method, device, equipment and medium

Abstract

The embodiment of the invention relates to the technical field of data processing, in particular to a green and ubiquitous distributed reasoning service method and device, equipment and a medium, and aims to provide green and ubiquitous distributed large model reasoning service. The method comprises the steps of adding received reasoning request groups into a task queue; executing the pre-filling tasks in sequence through the pre-filling instances; the decoding tasks of the models corresponding to the reasoning requests are added into a work list, the decoding tasks corresponding to the same group of reasoning requests are placed in the same batch, and the decoding tasks of each batch have corresponding preset execution time; circularly executing the decoding task in the work list through the decoding instance; key values generated in the task execution process are cached in the key value cache block pool; under the condition that the pre-filling task and the decoding task are executed, the corresponding model is switched, and the corresponding key value cache is loaded; and under the condition that all the decoding tasks are executed, obtaining a plurality of reasoning results corresponding to the plurality of groups of reasoning requests.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence technology, in particular, to a green ubiquitous distributed inference service method, device, equipment and medium. BACKGROUND

[0002] The inference service system is a service system specially constructed for large models. A large number of models are provided in the inference service system, and corresponding models are provided for different user requests to provide inference services. In the prior art, a large number of models are included in the inference service system, and at least one GPU (Graphics Processing Unit) instance is reserved for each service model in the system to ensure inference performance.

[0003] In the prior art, one model request is processed by one GPU instance for inference. For a new model request, inference must be performed after the previous request is executed and the GPU instance is released. In the case of insufficient GPU instances, the inference time of the model is likely to be too long, which reduces the inference efficiency of the system. SUMMARY

[0004] Embodiments of the present application provide a green ubiquitous distributed inference service method, device, equipment and medium, aiming to provide green ubiquitous distributed large model inference service.

[0005] The first aspect of the embodiments of the present application provides a green ubiquitous distributed inference service method, the method comprising: grouping and scheduling a plurality of received inference requests according to corresponding models to obtain a plurality of task queues corresponding to the plurality of inference requests; performing, by the pre-filling instance, pre-filling tasks of the models corresponding to each group of the inference requests in the task queue in sequence; adding decoding tasks of the models corresponding to the plurality of inference requests into a work list, and placing the decoding tasks corresponding to the same group of the inference requests in the same batch in the work list, each batch of the decoding tasks having a corresponding preset execution time; cyclically performing, by the decoding instance, the decoding tasks in the work list according to the execution time corresponding to each group of the decoding tasks; storing key-value caches generated by the pre-filling tasks and the decoding tasks corresponding to each group of the inference requests into a key-value cache block pool of the models corresponding to the inference requests; switching the corresponding models in the case of performing the pre-filling tasks and the decoding tasks corresponding to each group of the inference requests, and obtaining the key-value caches corresponding to the models from the key-value cache block pool in the case of performing the decoding tasks; In a case that each of the pre-filling tasks and each of the decoding tasks corresponding to each of the inference requests is executed, a single-step inference result corresponding to each of the inference requests is obtained.

[0006] Optionally, before each of the inference requests is executed by the pre-filling instance and the decoding instance, the method further comprises: Before the inference tasks are executed by the pre-filling instance and the decoding instance, a plurality of models corresponding to a plurality of the inference requests are determined. Components of the inference engine are loaded into the pre-filling instance and the decoding instance. According to the task queue and the work list, weight parameters of a model corresponding to a next group of the inference requests are pre-loaded.

[0007] Optionally, before each of the pre-filling tasks corresponding to each of the inference requests is executed by the pre-filling instance in sequence, the method further comprises: The inference requests using the same model are divided into the same group, and a plurality of groups of the inference requests are obtained. Each of the inference requests is put into a corresponding task queue according to a load of the pre-filling instance, and a plurality of task queues are obtained.

[0008] Optionally, before the decoding tasks in the work list are executed by the decoding instance in a cycle according to an execution time of each of the decoding tasks, the method further comprises: According to an execution time, a deadline, a model switching time and a service level objective corresponding to each of the decoding tasks, an execution time corresponding to each of the decoding tasks in the work list is determined.

[0009] Optionally, the decoding tasks in the work list are executed by the decoding instance in a cycle according to an execution time of each of the decoding tasks, comprising: A first batch of the decoding tasks in the work list is executed by the decoding instance. In a case that a preset time of executing the first batch of the decoding tasks exceeds the preset execution time, a current model is switched to a model corresponding to a next group of the inference requests in the work list. In a case that a round of the decoding tasks in the work list is executed, the decoding tasks in the work list are continuously executed from the beginning until all of the decoding tasks in the work list are executed.

[0010] Optionally, the key-value cache generated by each of the pre-filling tasks and each of the decoding tasks corresponding to each of the inference requests is stored into a key-value cache block pool of a model corresponding to each of the inference requests, comprising: determine an address of each key-value cache block in a key-value cache block pool of a model corresponding to the inference request; In a case where it is determined that there is a free block in the key-value cache block pool, send the key-value cache generated by the model to an address corresponding to the free block.

[0011] Optionally, in a case where each group of pre-filling and decoding tasks corresponding to the inference request is executed, the corresponding model is switched, and in a case where the decoding task is executed, the key-value cache corresponding to the model is loaded, comprising: In a case where each group of pre-filling tasks and decoding tasks corresponding to the inference request is executed, find the weight parameters of the model corresponding to the inference request in the model pool; In a case where the weight parameters are found, copy the weight parameters to the pre-filling instance and the decoding instance through a buffer; In a case where the decoding task is executed, obtain the key-value cache of the model corresponding to the inference request from the key-value cache block pool; Load the key-value cache into the decoding instance.

[0012] The second aspect of the embodiments of the present application provides a green and ubiquitous distributed inference service device, the device comprising: A request grouping module is configured to group and schedule a plurality of inference requests received according to corresponding models and the load of pre-filling instances, to obtain a plurality of task queues corresponding to a plurality of groups of the inference requests; A pre-filling execution module is configured to sequentially execute, by the pre-filling instance, the pre-filling tasks of the model corresponding to each group of the inference requests in the task queue; A work list establishing module is configured to add the decoding tasks of the model corresponding to a plurality of groups of the inference requests into a work list according to the load of decoding instances, and place the decoding tasks corresponding to the same group of the inference requests in the same batch in the work list, and each batch of the decoding tasks has a corresponding preset execution time; A decoding execution module is configured to cyclically execute the decoding tasks in the work list by the decoding instance according to the execution time corresponding to each group of the decoding tasks; A key-value cache module is configured to generate key-value caches of the pre-filling tasks and the decoding tasks corresponding to each group of the inference requests into a key-value cache block pool of the model corresponding to the inference request; A model switching module is configured to switch the corresponding model in a case where each group of pre-filling tasks and decoding tasks corresponding to the inference request is executed, and obtain the key-value cache corresponding to the model from the key-value cache block pool in a case where the decoding task is executed; A result obtaining module is configured to obtain a single-step inference result corresponding to the inference request in a case where each pre-filling task and each decoding task corresponding to each group of the inference requests are executed.

[0013] Optionally, the apparatus further comprises: A model determining module is configured to determine a plurality of models corresponding to a plurality of groups of the inference requests before the corresponding inference tasks are executed by the pre-filling instance and the decoding instance. A component loading module is configured to load components of an inference engine into the pre-filling instance and the decoding instance. A weight parameter loading module is configured to load weight parameters of a model corresponding to the inference request being processed into the pre-filling instance and the decoding instance. A model pre-fetching module is configured to pre-load weight parameters of a model corresponding to a next group of the inference requests according to the task queue and the work list.

[0014] Optionally, the request grouping module comprises: A request grouping sub-module is configured to group the inference requests using the same model into the same group to obtain a plurality of groups of the inference requests. A task queue establishing sub-module is configured to put each group of the inference requests into a corresponding task queue according to a load of the pre-filling instance to obtain a plurality of task queues.

[0015] Optionally, the apparatus further comprises: An execution time determining module is configured to determine an execution time of each group of the decoding tasks in the work list according to an execution time, a deadline, a model switching time and a service level objective corresponding to each decoding task.

[0016] Optionally, the decoding executing module comprises: A first decoding executing sub-module is configured to execute a first batch of decoding tasks in the work list by the decoding instance. A second decoding executing sub-module is configured to switch a current model to a model corresponding to a next group of the inference requests in the work list in a case where a preset time of executing the first batch of decoding tasks exceeds the preset execution time. A third decoding executing sub-module is configured to continue to execute decoding tasks in the work list from the beginning until all the decoding tasks in the work list are executed in a case where a round of the decoding tasks in the work list is executed.

[0017] Optionally, the key-value cache module comprises: The address determination sub-module is configured to determine the address of each key-value cache block in the key-value cache block pool of the model corresponding to the inference request. The key-value cache sending sub-module is configured to send the key-value cache generated by the model to the address corresponding to the idle block in the case where it is determined that there is an idle block in the key-value cache block pool.

[0018] Optionally, the model switching module comprises: The weight parameter searching sub-module is configured to search for the weight parameter of the model corresponding to the inference request in the model pool in the case where the pre-population task and the decoding task corresponding to each group of the inference request are executed. The parameter loading sub-module is configured to copy the weight parameter to the pre-population instance and the decoding instance through a buffer in the case where the weight parameter is searched for. The key-value cache obtaining sub-module is configured to obtain the key-value cache of the model corresponding to the inference request from the key-value cache block pool in the case where the decoding task is executed. The key-value cache loading sub-module is configured to load the key-value cache into the decoding instance.

[0019] The third aspect of the embodiments of the present application provides a readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in the method of the first aspect of the present application.

[0020] The fourth aspect of the embodiments of the present application provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method of the first aspect of the present application when executing the computer program.

[0021] The method provided in the application adopts the green ubiquitous distributed reasoning service method to group and schedule a plurality of reasoning requests according to the load of corresponding models and pre-filled instances, to obtain a plurality of task queues corresponding to the plurality of reasoning requests; the pre-filled instances execute the pre-filled tasks of the models corresponding to each group of reasoning requests in the task queue in sequence; the decoding tasks of the models corresponding to the plurality of reasoning requests are added to a work list according to the load of the decoding tasks, the decoding tasks corresponding to the same group of reasoning requests are placed in the same batch in the work list, and each batch of decoding tasks has a corresponding preset execution time; the decoding instances execute the decoding tasks in the work list in a cycle according to the execution time corresponding to each group of decoding tasks; the key-value caches generated by the pre-filled tasks and the decoding tasks corresponding to each group of reasoning requests are stored in the key-value cache block pool of the models corresponding to the reasoning requests; in the case of executing the pre-filled tasks and the decoding tasks corresponding to each group of reasoning requests, the corresponding models are switched, and in the case of executing the decoding tasks, the key-value caches corresponding to the models are obtained from the key-value cache block pool; in the case that the decoding tasks corresponding to the plurality of groups of reasoning requests are all executed, a plurality of reasoning results corresponding to the plurality of groups of reasoning requests are obtained.

[0022] In the method, a flag-level time-sharing multiplexing hardware resource multi-large language model reasoning system is constructed, the pre-filled tasks and the decoding tasks of the large language model are decoupled, the requests for the same model are grouped and scheduled, the pre-filled tasks are executed by the pre-filled instances to avoid frequent model switching, the decoding tasks are scheduled by batch polling to maximize the opportunity of time-sharing multiplexing of the decoding instances under the condition of meeting the target service time, the corresponding key-value cache block pool is set for each model to accelerate the caching and transmission of the key-value, and the reasoning tasks of multiple models are ensured to be completed within the target service time, the reasoning of the tasks is completed by multiple nodes in cooperation, the reasoning system can be accessed by various terminals, the resources of the server are saved, and the green ubiquitous distributed multi-large language model reasoning is realized. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the description of the embodiments of the application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0024] Figure 1 is a multi-model request distribution diagram within 24 hours; Figure 2 is a schematic diagram of the two-granularity resource multiplexing strategy proposed in an embodiment of the application; Figure 3 is a reasoning system workflow schematic diagram proposed by an embodiment of the present application; Figure 4 is a flowchart of a reasoning service method proposed by an embodiment of the present application; Figure 5 is a schematic diagram of a reasoning service device proposed by an embodiment of the present application; Figure 6 is a schematic diagram of an electronic device proposed by an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0026] Large language models (LLMS) are widely used today, and each model has different requirements for deployment and performance. The inference service of multiple large language models has higher requirements for the resource efficiency of the inference system. On the one hand, the service of each model needs to meet the token level deadline, which corresponds to two indicators: Time To First Token (TTFT), which measures the delay in the prefill stage of the large model, and Total Blocking Time (TBT), which measures the delay of each subsequent token generated in the decoding stage of the large model. These two indicators guarantee the overall service level objective of the model service, i.e., the target time spent by the inference service. On the other hand, the system for inference service presents a high skewedness distribution. Referring to Figure 1 , Figure 1 is a multi-model request distribution graph within 24 hours, as shown in Figure 1 , a large number of inference requests are concentrated on a few popular models, and the remaining requests are distributed among a series of low-frequency models. Low-frequency models account for 90% of the total number of models. In order to ensure inference performance, at least one inference instance needs to be reserved for each model. However, this strategy increases operating expenses (OPEX), and at least 17.7% of GPU resources need to be reserved to handle models that account for only 1.35% of requests, resulting in a serious decrease in resource utilization efficiency.

[0027] To achieve server-agnostic computing, existing research proposes to multiplex a single GPU instance to serve multiple models, including accelerating model weight loading to reduce the overhead of model switching, scheduling model requests to optimize the latency of each request, and selecting model parallel strategies and placement schemes to maximize multiplexing possibilities. These methods improve resource utilization by reserving instances only for active models (i.e., models that have at least one request being served). However, for models with low usage frequency in real loads, due to the long and unpredictable output length of large language model requests, a large number of models are in an active state even in the case of low request arrival efficiency. For example, in the case of a total request arrival rate of only 3.7 requests per second, a large number of models are in an active state. The defect of these systems is that they perform time-sharing multiplexing at the request granularity. In such systems, the execution of a new model request must wait for the completion of a previous request to release a GPU (Graphic Processing Unit) instance. If the number of reserved GPU instances is less than the number of active models, it will cause serious head-of-line blocking, which in turn causes the service level objective of the waiting model to be violated, i.e., the violation caused by not completing the inference service within the target service time.

[0028] A flag granularity solution is developed in the present application to overcome the limitations of existing methods, referring to Figure 2 , Figure 2 is a schematic diagram of two granularity resource multiplexing strategies proposed by an embodiment of the present application. The upper half is the resource multiplexing strategy in the related art, as shown in the figure, the method in the related art is prone to head-of-line blocking. The lower half is the resource multiplexing strategy proposed by the present application, where P represents a pre-filling task and D represents a decoding task. In the request granularity resource multiplexing strategy, the execution of a new model request must wait for the completion of a previous request to release a GPU instance. In the flag granularity resource multiplexing strategy proposed by the present application, flag granularity multiplexing allows the waiting request to preempt the running request after each flag is generated, thereby alleviating the head-of-line blocking phenomenon and significantly reducing the service level objective violation, i.e., exceeding the target service time.

[0029] Referring to Figure 3 , Figure 3 is a schematic diagram of the workflow of an inference system proposed by an embodiment of the present application, as Figure 3As shown, the inference service system mainly consists of three components: a proxy layer, a GPU pool (prefill instances and decode instances), and a memory management system. The proxy layer is responsible for receiving and scheduling inference requests, ensuring load balancing and guaranteeing system fault tolerance. State synchronization between the proxy layer and other components is achieved through a shared memory mechanism, such as using Redis (a key-value database). The system constructed in this application can schedule requests from different models to the same instance, such as... Figure 3 As shown in ①, ②, and ⑥ in the figure, the GPU pool, as a resource pool, contains a group of GPU instances virtualized by a cloud service provider. Each instance may contain one or more GPUs hosted on physical machines. In this application, a single instance performs either a pre-filling or decoding task (corresponding to ③, ⑤, and ⑧ in the figure) and can serve requests from different models according to the guidance of the flag-level scheduler. Model switching is a critical operation, as shown in ④, ⑦, and ⑨ in the figure. The Scheduler is the agent program. The memory manager coordinates the CPU (Central Processing Unit) memory and idle GPU memory of each node in the service cluster, mainly serving two subsystems: the first is the Quickloader (fast startup component), which uses available memory to cache model weights, accelerating the time to pull model weights from a remote model repository. The second is unified CPU swapping, which uses CPU memory to store swapped-out key-value caches.

[0030] refer to Figure 4 , Figure 4 This is a flowchart of a reasoning service method proposed in an embodiment of this application. Figure 4 As shown, the method includes the following steps: S11: The received multiple inference requests are grouped and scheduled according to the load of the corresponding model and pre-filled instance to obtain multiple task queues corresponding to the multiple groups of inference requests, and each task queue corresponds to a pre-filled instance.

[0031] In this embodiment, an inference request is a question or request submitted by a user to the inference service system that requires inference, such as answering a question or generating an image. The task queue contains pre-filled tasks corresponding to the inference requests, and pre-filled tasks corresponding to inference requests for the same model are in the same task queue.

[0032] In this embodiment, the inference service system maintains a task queue on each prefill instance, and each task queue is corresponding to the prefill tasks of the same group of requests. In the case of receiving multiple inference requests, first determine the model corresponding to each inference request, then group the inference requests for the same model, and add each group of inference requests to the corresponding task queue. Each task queue corresponds to a prefill instance.

[0033] In this embodiment, refer to the following code: Algorithm 1 Grouped Prefill-Phase Scheduling Input: Prefill instances Ip. 1:>Event: On arrival of request r: 2:min_load←oo 3: i*← Ip[0] 4: for all instance i in Ip do 5:for all group g in i.job_queue do 6:if g.model = r.model and g.size<MAX_GPSIZE then 7:Add r to group g # Prioritize existing groups 8:return 9:load ← Total time to execute all groups in i 10:if load<min_load then 11:min_load ← load # Pick the least loaded instance 12:i*←i 13:Make a group with r and append to i*.job_queue 14:>Event: On selecting a batch for prefll instance i execution: 15:Select one request from the front group of i.job_queue The above code is the core code of the grouping scheduling strategy, and the key idea is to group the requests for the same model to minimize the number of model switches. Each pre-populated instance maintains a task queue containing grouped tasks. When a new pre-populated task is received, the algorithm first adds the task to an existing group. If no suitable group is found, it means that the model corresponding to the request is different from the model corresponding to the previous group, which is a new model. A new group is then created and added to the task queue with the smallest load, where the load of the task queue is defined as the total time required to complete all pending groups, including task execution time and switching time. The batch size of the pre-populated instance can be limited to 1 (i.e., only one pre-populated task is executed from the front of the task queue each time), because the execution time of the pre-populated batch increases approximately linearly with the number of pre-populated flags. Small batches can reduce overall latency without significantly affecting throughput.

[0034] In this embodiment, the GPU pool is divided into two partitions, one for pre-population and the other for decoding. When a request is received, a pre-population instance in the pre-population partition is used to schedule pre-population tasks, and a decoding instance in the decoding partition is used to schedule subsequent decoding tasks. The first come first serve (FCFS) scheduling strategy is used to avoid frequent model switching and minimize the first flag time.

[0035] In this embodiment, the received multiple inference requests are grouped and scheduled according to the corresponding models to obtain multiple task queues corresponding to the multiple groups of inference requests. S11-1: Group the inference requests using the same model into the same group to obtain multiple groups of inference requests.

[0036] In this embodiment, when multiple model inference requests are received, the inference requests using the same model are grouped as the same group of inference requests, and then multiple groups of inference requests are obtained.

[0037] For example, 10 inference requests are received, where the first 5 inference requests require the use of model A, and the 6th to 10th inference requests require the use of model B. The first 5 inference requests are grouped as the first group of inference requests, and the 6th to 10th inference requests are grouped as the second group of inference requests.

[0038] S11-2: Place each group of inference requests in the corresponding task queue according to the load of the pre-populated instance to obtain multiple task queues.

[0039] In this embodiment, after grouping the received inference requests, each group of inference requests is placed into a corresponding task queue, and the load of the task queue corresponding to the pre-prepopulation instance is determined when placing, and the newly received inference request is preferentially placed into the task queue with smaller load.

[0040] S12: sequentially execute the pre-population task of the model corresponding to each group of inference requests in the task queue by the pre-population instance.

[0041] In this embodiment, the pre-population instance is an instance composed of multiple GPUs for executing the pre-population task of the large model, and the pre-population refers to pre-processing the input sequence and calculating the key intermediate state (such as Key-Value Cache of attention mechanism) before model inference, to provide cache data for subsequent decoding stage to reduce repeated calculation. For example, in a dialogue scenario, pre-population will encode the complete prompt (Prompt) of user input in advance to generate reusable intermediate tensors.

[0042] In this embodiment, first, the pre-population task of the model corresponding to each group of inference requests in the task queue is sequentially executed by the pre-population instance. In the GPU pool, the pre-population instance and the decoding instance are composed of different GPUs, and the pre-population instance is specially responsible for executing the pre-population task. When executing the pre-population task, the pre-population instance reads the first inference request in a group of inference requests, processes according to the input of the inference request, and obtains the corresponding key value. The pre-population instance has loaded the weight parameters of the corresponding model when executing the pre-population task.

[0043] S13: add the decoding tasks of the model corresponding to multiple groups of inference requests into multiple work lists according to the load of the decoding instance, each work list corresponding to a decoding instance, and the decoding tasks corresponding to the same group of inference requests are placed in the same batch in the work list, and each batch of decoding tasks has a corresponding preset execution time.

[0044] In this embodiment, the decoding task is to generate subsequent sequences step by step based on the key value output by the pre-population sequence. The work list is a list set in the decoding instance, which includes multiple batches of decoding tasks, and the preset execution time is the window time for executing each batch of decoding tasks.

[0045] In this embodiment, each decoding instance maintains a rotating work list, that is, each decoding instance corresponds to a work list, each batch in the list is composed of requests for the same model, and each batch is allocated a fixed time quota. After receiving multiple sets of inference requests, the decoding tasks of the model corresponding to each set of inference requests are placed in the work list, the decoding tasks corresponding to the same set of inference requests are placed in the same batch, each batch of decoding tasks has a corresponding preset execution time, and the tasks are preferentially placed in decoding instances with lower loads according to the loads of the decoding instances.

[0046] For example, in the work list, the preset execution time corresponding to the first set of inference requests is 2 seconds, and the preset execution time corresponding to the second set of inference requests is 3 seconds.

[0047] S14: The decoding instance circulates the decoding tasks in the work list according to the execution time corresponding to each set of decoding tasks.

[0048] In this embodiment, at the beginning of each round of inference, the execution time corresponding to the decoding tasks of each batch is calculated, and each batch of decoding tasks is executed according to the execution time. After the execution time of a batch of decoding tasks exceeds the preset execution time, the model is quickly switched and the decoding tasks of the next batch are executed, and the decoding tasks of multiple batches in the list are executed in this way until all the decoding tasks are executed.

[0049] In this embodiment, the decoding instance circulates the decoding tasks in the work list according to the execution time corresponding to each set of decoding tasks, including: S14-1: The decoding instance executes the decoding tasks of the first batch in the work list.

[0050] In this embodiment, when the decoding instance executes each round of decoding tasks, the decoding instance first executes the decoding tasks of the first batch in the work list, that is, the key values generated after the key values corresponding to the pre-filled tasks of the first batch of inference requests are obtained.

[0051] S14-2: In the case where the preset time for executing the decoding tasks of the first batch exceeds the preset execution time, the current model is switched to the model corresponding to the next set of inference requests in the work list.

[0052] In this embodiment, in the case where the preset time for executing the decoding tasks of the first batch exceeds the preset execution time, the current model is switched to the model corresponding to the next set of inference requests in the cache. Here, the GPU will pre-obtain the model corresponding to the decoding tasks of the next batch.

[0053] S14-3: In the case of performing a round of decoding tasks in the work list, continue to perform decoding tasks in the work list from the beginning until all decoding tasks in the work list are performed.

[0054] In this embodiment, in the case of performing a round of decoding tasks in the work list, continue to perform decoding tasks in the work list from the beginning until all decoding tasks in the work list are performed, and before each round of decoding task execution, the preset execution time of each batch of decoding tasks in the work list is calculated to ensure that the list is updated every round.

[0055] In this embodiment, the scheduling of decoding tasks is designed based on the uniqueness of large language model inference. The execution of decoding tasks is multi-round, and the output flag can be buffered. Let t be the time required for one decoding step, and d be the deadline of the step (i.e. the target flag interval TBT). For every n consecutive decoding steps, the request can tolerate a maximum delay of n (d-t) time, as long as the delay meets this time, it does not violate the service level target. Since t is usually small (e.g. tens of milliseconds), and d is relatively loose (e.g. for chat applications, TBT is 100 milliseconds), this part of the extra loose time can be used to serve other requests.

[0056] Reference the following code: Algorithm 2 Batched Decoding-Phase Scheduling Input: Decoding instances Ip. 1:>Event: On arrival of request r: 2:Dispatch as in Algorithm 1, deriving max batch sizes from the KVcache capacity on GPU, using work list sizes for load 3:>Event: Always on decoding instance i: 4: while True do 5:# Start of a round 6:Reorder i.work_list to group batches with the same model 7: Assign time quota q_k to the k-th batch in i.work_list 8: n_turn <- Number of batches in i.work_list 9: for all index k in 1 ~ n_turn do 10: # Start of a turn 11: Decode the k-th batch in i.work_list for q_k seconds The code is the core code of the round-robin scheduling scheme, and the idea is that each decoding instance maintains a rotating work list, each batch of decoding tasks in the list is composed of decoding tasks corresponding to requests for the same model, and is assigned a time quota, and new pre-filled completed requests are scheduled through a grouping mechanism, but the load is calculated by the size of the work list, and the batch size limit is determined by the key-value cache capacity on the GPU, and the execution of the work list is organized according to rounds.

[0057] Before the decoding instance executes the decoding tasks in the work list according to the execution time of each decoding task, the system determines the execution time of each decoding task in the work list according to the execution time of each decoding task, the deadline, the model switching time and the service level target. The algorithm allocates a time quota qi for the ith batch by the following formula: (1) Where c is the sum of the switching overheads of all unique models in the work list, and a is given by the following formula: (2) a is the inverse of the service level target achievement degree of the round. The algorithm will reorder the work list after each round of decoding task execution is completed to ensure that batches sharing the same model are placed adjacent to each other. This may occur when the key-value cache space required by a batch exceeds the available space on the GPU. Each batch will be decoded for qi time, and this formula ensures that , i.e. the quota assigned to the ith batch makes the allowable delay window time equal to a of the total round time. In this way, all decoding tasks in each round will always obtain a guaranteed service level target achievement degree, i.e. to ensure that each group of inference request corresponding decoding tasks are decoded within the target service time, thereby providing inference results to users.

[0058] S15: store the key-value cache generated by the pre-filling task and the decoding task corresponding to each group of inference requests into the key-value cache block pool of the model corresponding to the inference request.

[0059] In this embodiment, Key-Value Cache is a core technology for optimizing the inference efficiency of large language models. In generative tasks (such as text generation), the self-attention weight of the historical sequence needs to be calculated repeatedly each time decoding. KV Cache avoids repeated calculation of the historical sequence by caching the generated Key and Value. The key-value cache block pool is a cache block pool constructed based on the slab allocation mechanism. Each key-value cache block pool corresponds to a model, and the key-value shapes of each model may be different or the same.

[0060] In this embodiment, when performing the pre-filling task and the decoding task for each group of inference requests, key-value cache is generated. At this time, the key-value cache generated by the pre-filling task and the decoding task is stored in the key-value cache block pool of the model corresponding to the inference request.

[0061] In this embodiment, for storing key-value cache, slab allocation algorithm is used to realize unified storage in CPU memory to balance the memory waste caused by internal fragmentation and the time overhead caused by fine-grained memory management during memory allocation. For different shapes of cache blocks from different models, a unified CPU cache is managed through the slab allocation mechanism. The cache space is divided into fixed-size blocks, called slabs. Each slab is allocated to a specific key-value cache shape as a key-value cache block pool of that shape. When allocating memory, the system first uses the free blocks from the slabs of the same shape. If there are no free blocks available, the system will obtain a new slab for that shape. The system tracks each block by calculating the global offset from the starting position of the same CPU cache, and only considers the existence of the slab when allocating the block.

[0062] In this embodiment, the slab allocation mechanism balances between management overhead and internal fragmentation, and the degree of fragmentation is limited to the size of the slab. The slab allocation realizes efficient memory utilization under real workloads.

[0063] In this embodiment, the storing of the key-value cache generated by the pre-filling task and the decoding task corresponding to each group of inference requests into the key-value cache block pool of the model corresponding to the inference request includes: S15-1: determine the address of each key-value cache block in the key-value cache block pool of the model corresponding to the inference request.

[0064] In the embodiment, in the case that the key-value cache of the model corresponding to the inference request is generated, the address of the key-value cache block pool of the model corresponding to the inference request in the memory is determined. The position of the key-value cache block pool is a block area in the memory, which has a corresponding start address and a termination address.

[0065] S15-2: In the case that it is determined that there is a free block in the key-value cache block pool, the key-value cache generated by the model is sent to the address corresponding to the free block.

[0066] In the embodiment, after the address of the key-value cache block pool corresponding to the model is determined, it is determined whether there is a free block in the key-value cache block pool. In the case that there is a free block in the key-value cache block pool, the key-value cache generated by the model is sent to the corresponding key-value cache block pool.

[0067] S16: In the case that the pre-filling task and the decoding task corresponding to each group of inference requests are executed, the corresponding model is switched, and in the case that the decoding task is executed, the key-value cache corresponding to the model is obtained from the key-value cache block pool.

[0068] In the embodiment, in the case that the pre-filling task and the decoding task corresponding to each group of inference requests are executed, the model in the pre-filling instance and the decoding instance needs to be switched to the corresponding model, and the key-value cache already stored by the model is obtained from the key-value cache block pool, and then the inference task is executed.

[0069] In this embodiment, a high-speed model switching technique is used in the process of model switching. The high-speed model switching technique is based on the profiling of the real inference system running. Switching models on instances is not only limited by the delay of loading model weights from CPU memory to GPU, but also by the restart and initialization overhead of other components of the inference system. The following factors significantly affect the model switching time, including: Distributed executor: The inference engine supports model parallelism through distributed executors (such as Ray and NCCL). The initialization of these executors usually takes tens of seconds. Performance analysis and optimization: Some engines perform performance analysis and optimization (for example, deciding the size of the key-value cache or using CUDA graphs for computation acceleration), which also takes tens of seconds. Model weight loading: Loading model weights onto the GPU is a time-consuming process, and as the size of the model increases, the delay will be higher. Although modern PCIe buses provide high theoretical bandwidth (for example, the bandwidth of PCIe 4.0 is 32 GB / s), the inference engine often fails to fully utilize this bandwidth. For example, loading the LLaMA-13B model (using 2x tensor parallelism) through PCIe 4.0 takes about 4.6 seconds in a micro-benchmark, corresponding to an actual bandwidth of only 2.83 GB / s. Key-value cache initialization: Inference engines that use CPU memory to store key-value caches lock memory pages to improve data access performance. However, locking memory pages introduces a second-level overhead. Other components: The initialization of other components (such as schedulers, etc.) also affects the time overhead.

[0070] To address the above problems, the inference service system uses the method of pre-caching components to avoid repeated initialization of components other than models, and introduces model weight caching, pipeline loading, and model prefetching to speed up the loading of model weights.

[0071] In this embodiment, for the transmission of key-value caches, only in the following four cases will the block move synchronization operation be forced, which involves inherent data dependency relationships that model inference needs to wait for the block to be loaded onto the GPU; model switching needs to wait for the block to be swapped out to CPU memory; block move needs to wait for the source data to be ready; block move needs to wait for the target block to be emptied. The synchronization operation refers to the need to determine that the data transfer is complete after data transfer. Except for these four cases, no synchronization operation is needed, which ensures that the key-value cache transmission will not produce excessive delay.

[0072] In this embodiment, the movement of the block is efficiently synchronized by using CUDA Event (Compute Unified Device Architecture, Event), which is a core tool for implementing GPU asynchronous task management and performance tuning. On each instance GPU, the system maintains an independent CUDA stream, and each block movement is captured as a new Event by cudaEventRecord (used to insert an event into a specified CUDA stream and record the current GPU execution time). For the first and second cases described above, the instance will pause inference or model switching, and cudaEventSynchronize (used to force the host thread to wait for the specified CUDA event to complete recording, ensuring that the GPU operation associated with the event (such as the kernel function and data transfer) has been executed, and the relevant block movement is completed. For the third case, the instance will continue to call cudaStreamWaitEvent (a key API in CUDA for implementing inter-stream synchronization) to block only the relevant stream, allowing other CPU and GPU streams to continue execution. To ensure synchronization in the fourth case, the moving block and the corresponding Event are collected into a separate list, and a background thread is used to periodically poll the Event in the list using cudaEventQuery (a CUDA API for non-blocking Event state query), so that the moved block can be recycled to the idle list. Only when the background thread fails to recycle enough moving blocks before allocation, explicit blocking is used. This design effectively offloads the synchronization task of the fourth case to the background thread, reducing the overall overhead and thus reducing the overhead of key-value cache transmission.

[0073] In this embodiment, in the case of executing the pre-filling task and the decoding task corresponding to each group of inference requests, the corresponding model is switched, and in the case of executing the decoding task, the specific steps of obtaining the key-value cache corresponding to the model from the key-value cache block pool include: S16-1: In the case of executing the pre-filling task and the decoding task corresponding to each group of inference requests, the weight parameters of the model corresponding to the inference request are searched in the model pool.

[0074] In this embodiment, the model pool is a region in shared memory for storing weight parameters.

[0075] In this embodiment, in the case of executing the pre-filling task and the decoding task corresponding to each group of inference requests, the model corresponding to the group of inference requests is determined, and then the weight parameters of the model corresponding to the inference request are searched in the model pool.

[0076] S16-2: In the case of finding the weight parameter, the weight parameter is copied to the pre-filling instance and the decoding instance through the buffer.

[0077] In this embodiment, the buffer is a block of memory region unique to each GPU.

[0078] In this embodiment, the service system copies the weight parameter to the pre-filling instance and the decoding instance through the buffer in the case of finding the weight parameter in the model pool, that is, to the GPU processing the inference request. Further, the loading of the weight parameter is completed.

[0079] S16-3: In the case of executing the decoding task, the key-value cache of the model corresponding to the inference request is obtained from the key-value cache block pool.

[0080] In this embodiment, when processing the inference request, the key-value cache is generated, and the generated key-value cache is stored in the key-value cache block pool. In the process of cyclically executing the decoding task, in one window period, the decoding task corresponding to one inference request may not be completed, at which time the key-value cache needs to be stored, and the decoding task is continued to be executed when the next cycle arrives. In order to continue to execute the decoding task of the model, the key-value cache stored when the decoding task corresponding to the request is executed last time needs to be obtained from the corresponding key-value cache block pool to continue the decoding task last time.

[0081] S16-4: Load the key-value cache into the decoding instance.

[0082] In this embodiment, after obtaining the key-value cache generated by the model corresponding to the inference request when processing the decoding task last time, the key-value cache is loaded into the decoding instance before the decoding task starts to execute. The decoding instance runs the model corresponding to the inference request, and the model continues to execute the inference according to the obtained key-value cache.

[0083] S17: In the case where each group of pre-filling tasks corresponding to the inference request and each step decoding task are executed, the single-step inference result corresponding to the inference request is obtained.

[0084] In this embodiment, in the case where each group of pre-filling tasks corresponding to the inference request and each step decoding task are executed, a plurality of single-step inference results corresponding to a plurality of inference requests are obtained, that is, after each decoding task is completed, the inference result corresponding to the inference request is obtained.

[0085] For example, if the inference request is to generate a specific picture, the inference result is the generated picture, and if the inference request is to generate a text, the inference result is the generated text.

[0086] In this embodiment, a first-to-service scheduling strategy is designed for the pre-filling stage of the large language model, and a polling scheduling strategy is designed for the decoding stage. In the case of ensuring that the inference task does not violate the service level target, the processing of batch inference requests is realized, and each GPU can realize server-unaware model switching, thereby saving server resources. Through efficient key-value cache swap-in and swap-out technology, a unified CPU key-value cache storage mechanism and an asynchronous key-value cache transmission mechanism are designed to optimize the memory occupation of key-value cache storage and the delay overhead of transmission, thereby realizing efficient server-unaware model service and improving the resource utilization of the system.

[0087] In another embodiment of the present application, before sequentially executing the pre-filling tasks corresponding to each group of inference requests by the pre-filling instance, the method further comprises: S21: Before executing the corresponding inference tasks by the pre-filling instance and the decoding instance, a plurality of models corresponding to a plurality of groups of inference requests are determined.

[0088] In this embodiment, different inference requests are processed by different models. When the inference service system receives an inference request sent by a user, the inference request is parsed to obtain the content to be inferred by the inference request and the model required by the inference request. Each group of inference requests in a plurality of groups of inference requests corresponds to a model.

[0089] S22: Loading the components of the inference engine into the pre-filling instance and the decoding instance.

[0090] In this embodiment, the components of the large model are some basic components that make up the large model. Different large models have the same components.

[0091] In this embodiment, the service system initializes the inference engine only once on each instance and caches the components other than the model and the key-value cache. When initializing the inference engine, the components corresponding to the inference engine are loaded into the pre-filling instance and the decoding instance.

[0092] For example, the components are universal components such as decoders, encoders, and attention mechanisms.

[0093] In this embodiment, by pre-caching the components of the model in the instance, only the weights and other parameters of each model need to be obtained when switching the model, so that the loading of universal components does not occupy the model switching time, accelerating the switching of the model.

[0094] In addition, the service system performs performance analysis on each model in advance, such as analysis of the model key value cache size, and through the analysis, the switching loading time of each model can be determined. Through the unified CPU cache, the need for locking the memory page in the model switching process is eliminated.

[0095] S23: load the weight parameters of the model corresponding to the current processing inference request into the pre-populated instance and the decoding instance.

[0096] In this embodiment, before performing the inference task corresponding to the inference request, the weight parameters of the model corresponding to the current inference request are loaded into the pre-populated instance and the decoding instance, and the loading of the model is completed.

[0097] In this embodiment, a quickloader (dynamic loading component) is introduced, which is composed of a model pool in a shared memory and a lock buffer unique to each GPU, and the model cache pool is from the original tensor block in the checkpoint file. In the case of a model switching request issued by the service system, the quickloader first checks the required weight in the model pool, and if the corresponding weight is not found in the model pool, it is pulled from the remote model repository. After obtaining the weight, the locked buffer is copied to the GPU, and this copying process is multi-threaded, block-based, and pipelined, which fully utilizes the PCIe bandwidth and asynchronous data transmission, thereby accelerating the model switching. For the cached models in the model pool, the system constructed can achieve a model loading speed of seconds.

[0098] S24: Preload the weight parameters of the next group of models corresponding to the inference requests according to the task queue and the work list.

[0099] In this embodiment, the pre-populated tasks corresponding to each group of inference requests are stored in the task queue, and the decoding tasks corresponding to each group of inference requests are stored in the work list. These tasks are arranged in order, and according to the task queue and the work list, the model corresponding to the next group of inference requests can be determined. After the model corresponding to the next group of inference requests is determined, the corresponding weight parameters are obtained from the model pool, and the weight parameters are preloaded to reduce the delay of model switching.

[0100] In this embodiment, in order to further reduce the overhead of model switching, the next required model is pre-fetched based on the pre-populated task queue and the decoding work list, and data transmission and GPU calculation are overlapped. This method is particularly effective for decoding instances, because the time quota of each round usually completely masks the time overhead of switching, thereby realizing instant switching without the awareness of the server.

[0101] In the above embodiments of the present application, a service level target aware flag level request scheduling technology, a high-speed model switching technology, and an efficient key-value cache swap-in and swap-out technology are proposed. On the one hand, a first-come-first-execution algorithm and a polling scheduling algorithm are respectively designed for the pre-filling and decoding tasks of a large language model, the inference requests for the same model are divided into a group to perform the pre-filling task, and the decoding task corresponding to each group of inference requests is executed in a batch-by-batch round-robin manner, which ensures that multiple inference requests are processed at the same time under the condition that the service level target is not violated, and improves the overall efficiency of the system. The high-speed model switching is realized through the cache component, the fast model loading, and the model prefetching. Through the unified CPU key-value cache storage mechanism and the asynchronous key-value cache transmission mechanism, the memory occupation and transmission delay of the key-value cache storage are optimized, the efficient server unaware model service is provided, and the server resources are saved. This method is suitable for various platforms and terminals, and realizes the green ubiquitous distributed inference service.

[0102] Based on the same inventive concept, an embodiment of the present application provides a green ubiquitous distributed inference service device. Referring to Figure 5 , Figure 5 is a schematic diagram of an inference service device 500 according to an embodiment of the present application. As shown in Figure 5 , the device comprises: a request grouping module 501 configured to group and schedule a plurality of inference requests according to corresponding models to obtain a plurality of task queues corresponding to the plurality of inference requests; a pre-filling execution module 502 configured to sequentially execute, by the pre-filling instance, a pre-filling task of a model corresponding to each group of inference requests in the task queue; a work list establishing module 503 configured to add decoding tasks of the models corresponding to the plurality of groups of inference requests into a work list, wherein the decoding tasks corresponding to the same group of inference requests are placed in the same batch in the work list, and each batch of decoding tasks has a corresponding preset execution time; a decoding execution module 504 configured to cyclically execute, by the decoding instance, the decoding tasks in the work list according to the execution time corresponding to each group of decoding tasks; a key-value cache module 505 configured to store key-value caches generated by the pre-filling tasks and the decoding tasks corresponding to each group of inference requests into a key-value cache block pool of the model corresponding to the inference requests; a model switching module 506 configured to switch the corresponding model in the case of executing the pre-filling tasks and the decoding tasks corresponding to each group of inference requests, and acquire the key-value cache corresponding to the model from the key-value cache block pool in the case of executing the decoding tasks; The result obtaining module 507 is configured to obtain a plurality of inference results corresponding to a plurality of inference requests in a case where a plurality of decoding tasks corresponding to the plurality of inference requests are all executed.

[0103] Optionally, the apparatus further comprises: The model determining module is configured to determine a plurality of models corresponding to a plurality of inference requests before the corresponding inference tasks are executed by the pre-populated instance and the decoding instance. The component loading module is configured to load components of an inference engine into the pre-populated instance and the decoding instance. The weight parameter loading module is configured to load weight parameters of a model corresponding to the inference request being processed into the pre-populated instance and the decoding instance. The model caching module is configured to pre-load weight parameters of a model corresponding to a next group of inference requests according to the task queue and the work list.

[0104] Optionally, the request grouping module comprises: The request grouping sub-module is configured to group the inference requests using the same model into the same group to obtain a plurality of groups of inference requests. The task queue establishing sub-module is configured to put each group of inference requests into a corresponding task queue according to a load of the pre-populated instance to obtain a plurality of task queues.

[0105] Optionally, the apparatus further comprises: The execution time determining module is configured to determine an execution time of each group of decoding tasks in the work list according to an execution time, a deadline, a model switching time, and a service level objective corresponding to each decoding task.

[0106] Optionally, the decoding execution module comprises: The first decoding execution sub-module is configured to execute a first batch of decoding tasks in the work list by the decoding instance. The second decoding execution sub-module is configured to switch a current model to a model corresponding to a next group of inference requests in the work list in a case where a preset time of executing the first batch of decoding tasks exceeds the preset execution time. The third decoding execution sub-module is configured to continue to execute decoding tasks in the work list from the beginning in a case where a round of the decoding tasks in the work list is executed, until all the decoding tasks in the work list are executed.

[0107] Optionally, the key-value cache module comprises: The address determination sub-module is configured to determine the address of each key-value cache block in the key-value cache block pool of the model corresponding to the inference request. The key-value sending sub-module is configured to send the key-value cache generated by the model to the address corresponding to the idle block in the case where it is determined that there is an idle block in the key-value cache block pool.

[0108] Optionally, the model switching module comprises: The weight parameter searching sub-module is configured to search for the weight parameter of the model corresponding to the inference request in the model pool in the case where the pre-population task and the decoding task corresponding to each group of inference requests are performed. The parameter loading sub-module is configured to copy the weight parameter to the pre-population instance and the decoding instance through a buffer in the case where the weight parameter is searched for. The key-value obtaining sub-module is configured to obtain the key-value cache of the model corresponding to the inference request from the key-value cache block pool in the case where the decoding task is performed. The key-value loading sub-module is configured to load the key-value cache into the decoding instance.

[0109] Based on the same inventive concept, another embodiment of the present application provides a readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps in the inference service method according to any one of the above embodiments of the present application.

[0110] Based on the same inventive concept, another embodiment of the present application provides an electronic device, Figure 6 is a schematic diagram of an electronic device 600 according to an embodiment of the present application, which comprises a memory 602, a processor 601, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the inference service method according to any one of the above embodiments of the present application when executed.

[0111] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts are described in the part of the method embodiment.

[0112] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts between the embodiments can be referred to each other.

[0113] Those skilled in the art will appreciate that embodiments of the application can be supplied as a method, a device, or a computer program product. Thus, embodiments of the application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0114] Embodiments of the application are described herein with reference to the drawings, in which are shown flowcharts and / or block diagrams of methods, apparatuses (systems) and computer program products according to embodiments of the application. It will be understood that each flow and / or block of the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing terminal apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal apparatus, create means for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more functions specified by one or more blocks Figure 1 one or more functions specified by one or more blocks.

[0115] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing terminal apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowcharts and / or block diagrams block or blocks. Figure 1 one or more functions specified by one or more blocks Figure 1 one or more functions specified by one or more blocks.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal apparatus to cause a series of operational steps to be performed on the computer or other programmable terminal apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal apparatus provide steps for implementing the flowcharts and / or block diagrams block or blocks. Figure 1 one or more functions specified by one or more blocks Figure 1 one or more functions specified by one or more blocks.

[0117] While preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the foregoing description. Therefore, the appended claims are intended to cover all such variations and modifications as falling within the scope of the application.

[0118] Finally, it is to be understood that the phraseology or terminology such as "first" and "second" etc. used herein is merely intended to differentiate one entity or operation from another entity or operation, without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0119] The above provides a detailed introduction to the green ubiquitous distributed reasoning service method, device, equipment and medium provided by the present application. The principles and implementation modes of the present application are described by applying specific examples. The above example description is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A green pervasive distributed inference service method, characterized in that, The method comprises: grouping and scheduling the received plurality of inference requests according to the load of corresponding models and pre-filling instances to obtain a plurality of task queues corresponding to a plurality of groups of the inference requests, each of the task queues corresponding to a pre-filling instance; sequentially executing, by the pre-filling instance, a pre-filling task of the model corresponding to each group of the inference requests in the task queue; adding decoding tasks of the model corresponding to a plurality of groups of the inference requests into a plurality of work lists according to the load of decoding instances, each of the work lists corresponding to a decoding instance, the decoding tasks corresponding to the same group of the inference requests being placed in the same batch in the work list, and each batch of the decoding tasks having a corresponding preset execution time; cyclically executing, by the decoding instance, the decoding tasks in the work list according to the execution time corresponding to each group of the decoding tasks; storing key-value caches generated by the pre-filling task and the decoding task corresponding to each group of the inference requests into a key-value cache block pool of the model corresponding to the inference requests; switching the corresponding model in the case of executing the pre-filling task and the decoding task corresponding to each group of the inference requests, and obtaining the key-value cache corresponding to the model from the key-value cache block pool in the case of executing the decoding task; obtaining a single-step inference result corresponding to the inference request in the case of completing the execution of the pre-filling task and each decoding task corresponding to each group of the inference requests.

2. The green pervasive distributed inference service method of claim 1, wherein, The method further comprises: determining a plurality of models corresponding to a plurality of groups of the inference requests before executing the corresponding inference tasks by the pre-filling instance and the decoding instance; loading components of an inference engine into the pre-filling instance and the decoding instance; loading weight parameters of the model corresponding to the inference request currently being processed into the pre-filling instance and the decoding instance; preloading weight parameters of the model corresponding to the next group of the inference requests according to the task queue and the work list.

3. The green pervasive distributed inference service method of claim 1, wherein, The grouping and scheduling the received plurality of inference requests according to the load of corresponding models and pre-filling instances to obtain a plurality of task queues corresponding to a plurality of groups of the inference requests comprises: grouping the inference requests using the same model into the same group to obtain a plurality of groups of the inference requests; putting each group of the inference requests into the corresponding task queue according to the load of the pre-filling instance to obtain a plurality of groups of the task queues.

4. The green pervasive distributed inference service method of claim 1, wherein, Before cyclically executing, by the decoding instance, the decoding tasks in the work list according to the execution time corresponding to each group of the decoding tasks, the method further comprises: determining the execution time corresponding to each group of the decoding tasks in the work list according to the execution time, the deadline, the model switching time and the service level target corresponding to each of the decoding tasks.

5. The green pervasive distributed inference service method of claim 1, wherein, The cyclically executing, by the decoding instance, the decoding tasks in the work list according to the execution time corresponding to each group of the decoding tasks comprises: executing, by the decoding instance, the decoding tasks in the first batch in the work list; switching the current model to a next model corresponding to a next group of the inference requests in the work list in a case that a preset time for performing the decoding task of the first batch exceeds the preset execution time; continuing to perform the decoding task in the work list from the beginning in a case that a round of the decoding task in the work list is performed, until all the decoding tasks in the work list are performed.

6. The green pervasive distributed inference service method of claim 1, wherein, The key-value cache generated by each group of the pre-filling task and the decoding task corresponding to the inference request is stored into a key-value cache block pool of the model corresponding to the inference request, including: determining an address of each key-value cache block in the key-value cache block pool of the model corresponding to the inference request; in a case that there is a free block in the key-value cache block pool, sending the key-value cache generated by the model to an address corresponding to the free block.

7. The green pervasive distributed inference service method of claim 1, wherein, The switching of the corresponding model in a case that each group of the pre-filling task and the decoding task corresponding to the inference request is performed, and the acquisition of the key-value cache corresponding to the model from the key-value cache block pool in a case that the decoding task is performed, including: finding a weight parameter of the model corresponding to the inference request in a model pool in a case that each group of the pre-filling task and the decoding task corresponding to the inference request is performed; copying the weight parameter to the pre-filling instance and the decoding instance through a buffer in a case that the weight parameter is found; acquiring the key-value cache of the model corresponding to the inference request from the key-value cache block pool in a case that the decoding task is performed; loading the key-value cache into the decoding instance.

8. A green pervasive distributed inference service apparatus, characterized by, The apparatus includes: a request grouping module configured to group and schedule a plurality of inference requests received according to a load of a corresponding model and a pre-filling instance, to obtain a plurality of task queues corresponding to a plurality of groups of the inference requests, each of the task queues corresponding to a pre-filling instance; a pre-filling execution module configured to sequentially perform, by the pre-filling instance, a pre-filling task of the model corresponding to each group of the inference requests in the task queue; a work list establishing module configured to add decoding tasks of the models corresponding to a plurality of groups of the inference requests into a plurality of work lists according to a load of a decoding instance, each of the work lists corresponding to a decoding instance, the decoding tasks corresponding to a same group of the inference requests being placed in a same batch in the work list, each batch of the decoding tasks having a corresponding preset execution time; a decoding execution module configured to cyclically perform, by the decoding instance, the decoding tasks in the work list according to an execution time corresponding to each group of the decoding tasks; a key-value cache module configured to store key-value caches generated by each group of the pre-filling task and the decoding task corresponding to the inference request into a key-value cache block pool of the model corresponding to the inference request; a model switching module configured to switch the corresponding model in a case that each group of the pre-filling task and the decoding task corresponding to the inference request is performed, and to acquire the key-value cache corresponding to the model from the key-value cache block pool in a case that the decoding task is performed. A result obtaining module is configured to obtain a single-step inference result corresponding to the inference request when each pre-filled task corresponding to the inference request and each step decoding task are executed.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the steps in the method of any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps in the method of any one of claims 1 to 7.

Images