Scheduling method and device of reasoning task, computing equipment and program product
By accurately predicting the execution cost of the accelerator and dynamically scheduling the inference tasks of the neural network model, the problems of insufficient resource utilization and poor user experience in the existing technology are solved, and efficient resource utilization and improved user interaction experience are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2026-03-10
AI Technical Summary
Existing technologies cannot fully utilize device computing resources in neural network model inference task scheduling, resulting in a decline in user experience. In particular, when faced with multiple concurrent requests, scheduling methods based on fixed priorities or empirical predictions cannot guarantee efficient utilization of hardware resources and user interaction experience.
By accurately predicting the execution cost of each inference task performed by the accelerator, the computing device flexibly schedules the target inference tasks, ensuring full utilization of the accelerator's computing resources and improving the user experience. Dynamic scheduling is achieved by using a method that correlates the execution cost with the execution progress, priority, waiting time, accelerator hardware parameters, and neural network model feature parameters.
It achieves efficient utilization of accelerator computing resources, improves inference efficiency, avoids long user wait times, and enhances user experience.
Smart Images
Figure CN121636073A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer, in particular to a scheduling method and device of inference tasks, a computing device and a program product. BACKGROUND
[0002] In the process of using a neural network model for inference, usually involves tasks in different stages, for example, when a large language model (LLM) infers according to a prompt input by a user, it can be divided into two different stages of tasks, the first is a prefill task: the LLM generates the first token in the answer according to the prompt; the second is a decode task: the LLM uses the information generated in the prefill stage, and then the first token continues to generate subsequent tokens to obtain a complete inference result.
[0003] When the powerful inference function of a neural network model is promoted as a business, it usually faces challenges such as multiple request concurrency, and needs to reasonably schedule devices to perform inference tasks in different stages through an efficient task scheduling method. However, the related technology usually fixes the priority of executing a certain task, or schedules according to the experience of inferring the time required by the task, which cannot fully utilize the computing resources of the device, and cannot guarantee the user experience. SUMMARY
[0004] The present application provides a scheduling method and device of inference tasks, a computing device and a program product, which can accurately estimate the execution cost of the accelerator executing each inference task, and can realize flexible and reasonable scheduling of each inference task, so as to fully utilize the computing resources of the accelerator, improve the inference efficiency, and improve the user experience.
[0005] In a first aspect, the present application provides a scheduling method of inference tasks, the method comprising: a computing device determining an execution cost of an accelerator executing each inference task, the execution cost being related to the execution progress of the inference task; wherein each inference task is a task executed by different running stages in a neural network model; the computing device determining a target inference task from each inference task according to the execution cost; and the computing device instructing the accelerator to execute the target inference task.
[0006] It can be understood that the computing device can accurately estimate the performance-price ratio of the accelerator executing each inference task by determining the execution cost, and then the computing device instructs the accelerator to execute the target inference task according to the execution cost, realizes flexible and accurate dynamic scheduling, can guarantee that the computing resources of the accelerator are fully utilized, improve the inference efficiency, and can avoid that the user waits for the inference result for a long time, and improve the user experience.
[0007] In a possible implementation, the execution cost is related to a priority of each inference task, or a waiting time of each inference task, or a hardware parameter of the accelerator, or a characteristic parameter of the neural network model running on the accelerator.
[0008] It can be understood that the parameters in different dimensions can improve the accuracy of determining the execution cost, and thus help the computing device improve the accuracy of determining the target inference task and improve the inference efficiency.
[0009] In a possible implementation, the computing device determines the execution cost of the accelerator executing each inference task, including: the computing device determines an execution time of the accelerator executing each inference task; and the computing device determines the execution cost according to the execution time of each inference task.
[0010] It can be understood that the execution cost determined by the computing device according to the execution time of the accelerator executing each inference task can improve the accuracy of the execution cost, and help the computing device schedule the accelerator to preferentially execute the inference task with shorter execution time, thereby improving the inference efficiency.
[0011] In a possible implementation, the computing device determines the execution time of the accelerator executing each inference task, including: the computing device determines operators corresponding to each inference task and an execution order between the operators according to a characteristic parameter of the neural network model; the computing device predicts execution time of each operator according to a hardware parameter of the accelerator and an execution progress of each inference task; and the computing device determines the execution time of the accelerator executing each inference task according to the execution order and the execution time of each operator.
[0012] It can be understood that, since the hardware resources of the accelerator required by inference tasks with different execution progress are different, the computing device can improve the accuracy of estimating the execution time of the accelerator executing the operators corresponding to each inference task according to the hardware parameter of the accelerator and the execution progress of each inference task, and thus improve the accuracy of the computing device estimating the execution time of the accelerator executing each inference task according to the execution time of each operator and the execution order between the operators, for example, the accelerator executes operator A first, then executes operator B, and executes operator C in parallel when executing operator B.
[0013] In a possible implementation, the computing device determines the execution cost of the accelerator executing each inference task, including: the computing device determines the execution cost of the accelerator executing each inference task based on an execution cost calculation model, which is a mathematical model constructed according to the hardware parameter of the accelerator and the characteristic parameter of the neural network model running on the accelerator.
[0014] It can be understood that since the execution cost calculation model is constructed according to the hardware parameters of the accelerator and the characteristic parameters of the neural network model running on the accelerator, that is, a corresponding execution cost calculation model can be determined according to different characteristic parameters or hardware parameters, so that the computing device can be ensured not to be affected by changes in the characteristic parameters of the neural network model or the hardware parameters of the accelerator when determining the execution cost, thereby significantly improving the universality. Even if the characteristic parameters of the neural network model or the hardware parameters of the accelerator change, the computing device can still accurately determine the execution cost, and then implement flexible and reasonable scheduling.
[0015] In a possible implementation, based on the execution cost calculation model, the execution cost of the accelerator executing each inference task is determined, including: inputting the execution progress of each inference task into the execution cost calculation model to obtain the execution cost of the accelerator executing each inference task.
[0016] It can be understood that based on the execution cost calculation model, the execution progress of each inference task is inputted as input, and the execution cost of the accelerator executing each inference task can be accurately calculated, thereby helping the computing device to improve the accuracy of scheduling the accelerator and improve the inference efficiency.
[0017] In a possible implementation, the execution cost of the accelerator executing each inference task is determined, including: determining the execution cost of the accelerator executing each inference task according to the priority of each inference task.
[0018] It can be understood that according to the priority of each inference task, the execution cost (such as the higher the priority of the inference task, the lower the corresponding execution cost) is determined, which can avoid the inference task with high priority waiting for a long time, avoid some users waiting for the inference result for a long time, improve the user experience, and through the priority and execution time, the accuracy of determining the execution cost can be further improved.
[0019] In a possible implementation, the execution cost of the accelerator executing each inference task is determined, including: determining the execution cost of the accelerator executing each inference task according to the waiting time of each inference task.
[0020] It can be understood that according to the waiting time of each inference task, the execution cost (such as the longer the waiting time of the inference task, the lower the corresponding execution cost) is determined, which can avoid the inference task waiting for a long time, avoid the user waiting for the inference result for a long time, improve the user experience, and through the waiting time, the priority and the execution time, the accuracy of determining the execution cost can be further improved.
[0021] In a possible implementation, the neural network model is a large language model (LLM), and the inference tasks include prefill tasks and decode tasks.
[0022] It can be understood that the computing device can flexibly and reasonably schedule the two different stages of inference tasks by determining the execution cost of the accelerator to perform the prefill tasks and the decode tasks, so as to fully utilize the computing resources of the accelerator, improve the inference efficiency, and improve the user experience.
[0023] In a possible implementation, according to the execution cost, the computing device determines a target inference task from the inference tasks, including: the computing device determines a task with an execution cost less than or equal to a specified threshold as the target inference task.
[0024] It can be understood that the computing device preferentially schedules the inference task with a smaller execution cost of the accelerator, which can improve the inference efficiency.
[0025] In a possible implementation, the specified threshold is related to the execution cost of the inference tasks.
[0026] It can be understood that the specified threshold is determined according to the execution cost of the inference tasks, which can improve the accuracy of the computing device to determine the target inference task, and thus ensure the rationality of the scheduling.
[0027] In a possible implementation, the computing device instructs the accelerator to perform the target inference task, including: the computing device generates a target instruction and sends it to the accelerator, and the target instruction is used to instruct the accelerator to perform the target inference task.
[0028] It can be understood that the computing device can ensure that the accelerator performs the target inference task according to the scheduling of the computing device by generating the target instruction and sending it to the accelerator.
[0029] In a second aspect, the present application provides a scheduling device for an inference task.
[0030] In a possible implementation, the present application can divide the scheduling device for the inference task into functional modules according to the method provided in the first aspect. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. For example, the present application can divide the scheduling device for the inference task into a first determining module, a second determining module, and an instructing module according to functions. The possible technical solutions and beneficial effects of each functional module described above can be referred to the technical solutions provided in the first aspect or the corresponding possible implementation thereof, which will not be described here.
[0031] In a third aspect, an embodiment of the present application provides a computing device, comprising a processor and a memory, the processor being coupled to the memory; the memory is configured to store computer instructions, the computer instructions being loaded and executed by the processor to enable the computing device to implement the inference task scheduling method according to the above aspect.
[0032] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing at least one computer program instruction, the computer program instruction being loaded and executed by a processor to implement the inference task scheduling method according to the above aspect.
[0033] In a fifth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising computer instructions stored in a computer readable storage medium. A processor of a computing device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computing device to perform the inference task scheduling method provided in various optional implementation manners of the first aspect.
[0034] The specific description of the second aspect to the fifth aspect and various implementation manners thereof in the present application can refer to the detailed description in the first aspect and various implementation manners thereof; and the beneficial effects of the second aspect to the fifth aspect and various implementation manners thereof can refer to the beneficial effect analysis in the first aspect and various implementation manners thereof, which will not be described herein again.
[0035] These aspects or other aspects of the present application will be more apparent in the following description. BRIEF DESCRIPTION OF DRAWINGS
[0036] Figure 1 A schematic diagram of an exemplary application scenario provided by an embodiment of the present application;
[0037] Figure 2 A structural schematic diagram of a computing device provided by an embodiment of the present application;
[0038] Figure 3 A flowchart of an inference task scheduling method provided by an embodiment of the present application;
[0039] Figure 4 A flowchart of an inference task scheduling method provided by an embodiment of the present application;
[0040] Figure 5 A schematic diagram of an operator flowchart related to an embodiment shown in the figure; Figure 4
[0041] Figure 6 Another flowchart of a scheduling method of a reasoning task provided by an embodiment of the present application is shown in FIG. 2.
[0042] Figure 7 A structure diagram of a scheduling device of a reasoning task provided by an embodiment of the present application is shown in FIG. 3. DETAILED DESCRIPTION
[0043] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.
[0044] In the present disclosure, "a plurality of" means two or more. "And / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.
[0045] In addition, in the description of the embodiments of the present application, "a plurality of" means two or more, unless otherwise specified. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.
[0046] In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, "first", "second", and the like are used to distinguish the same items or similar items with basically the same function and effect. Those skilled in the art can understand that "first", "second", and the like do not limit the quantity and execution order, and "first", "second", and the like do not necessarily mean different. At the same time, in the embodiments of the present application, "exemplary" or "for example" is used to represent an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, "exemplary" or "for example" is used to present the relevant concept in a specific way, for understanding.
[0047] First, the application scenarios of the embodiments of the present application are exemplarily introduced.
[0048] The application of current neural network models has penetrated into many scenarios, such as image / audio generation applications based on generator adversarial networks (GAN) models, and intelligent question answering systems based on large language models (LLM), and so on.
[0049] When providing users with inference services based on neural network models, limited system resources often face a large number of tasks to be processed, including not only a large number of new requests input by users, but also tasks at different stages of the inference process. Reasonable scheduling methods are needed to fully utilize computing resources, improve inference efficiency, and shorten the time for users to wait for inference results.
[0050] Taking an intelligent question answering system based on a large language model (LLM) as an example, in the inference process of the LLM, tasks can be divided into two different stages. The first is the prefill task: the LLM generates the first token in the answer according to the prompt input by the user. The second is the decode task: the LLM uses the information generated in the prefill stage to continue generating subsequent tokens after the first token, obtaining the complete inference result.
[0051] For example, the prompt input by the user is "What is the speed of light?" The corresponding prefill task can be for the LLM to generate the first token "speed of light" in the inference result based on the prompt. After that, the prefill task for this request ends, and the next stage task, i.e., the decode task, needs to be performed. The decode task can be for the LLM to use the information generated in the prefill stage to continue generating the next token "about" after the first token "speed of light". The LLM continues to perform the decode task, which can take the newly generated token "about" as input, combined with the relevant information generated by the model during the inference of the token "about", to continue generating the next token "for". In this way, the LLM performs multiple rounds of decode tasks until the complete inference result "speed of light is about 300 million kilometers per second" is obtained.
[0052] In an intelligent question answering system based on an LLM, there are often multiple requests for prefill tasks to be performed, as well as multiple decode tasks that have not been completely executed. This requires the system not only to start the prefill task for the new prompt input by the user in a timely manner, but also to consider the decode task for the results of the prefill task performed previously, to avoid long waiting time for the complete inference result by the user.
[0053] In the traditional static scheduling method, multiple user requests can be processed in the same batch, in other words, multiple user requests will be input to the LLM for inference as a larger request, which can improve the utilization of system computing resources, and thus achieve greater throughput. However, in this method, since multiple requests are inferred as the same batch, and the length of the inference result is often different, during the inference process, the requests that have already finished inference also need to wait for all the other requests in the same batch to finish inference before returning to the user, resulting in that some users cannot receive complete answers for a long time.
[0054] To improve this problem, the related technology proposes a dynamic scheduling method, such as continuous batching technology, which can continuously add user requests to the current batch without reaching the set maximum inference batch, thereby ensuring that the prefilled or decoded batch reaches the maximum of the system, thereby fully utilizing the computing resources. Moreover, in the continuous batching technology, during the decoding process of the user's request, if the request is inferred to the last token, the decoding result of the request can be returned to the user first. In this method, after returning the decoding result to the user, the remaining requests to be decoded can be suspended first, and then the prefilled and decoded tasks are scheduled.
[0055] Specifically, the existing dynamic scheduling method has the following two scheduling strategies when scheduling prefilled and decoded tasks:
[0056] 1) Prioritize scheduling prefilled tasks.
[0057] This method can include the following 4 steps: S110, receiving an inference request. S120, determine whether it meets the condition of "decode queue is not full and there is available video memory space", if yes, execute step S130; if no, execute step S140. S130, execute the prefilled task. S140, execute the decoding task. In step S120, for example, the system sets the maximum value of the decoding queue to 10 requests, and there are already 8 requests in the current decoding queue, so it can be determined that the decoding queue is not full, and the current free video memory space is 8 gigabytes (GB), which is greater than the required 2 GB, so it can be determined that there is available video memory space, and then step S130 is executed.
[0058] The prefill-priority scheduling method can cause the system to frequently perform the prefill task, thereby prolonging the waiting time of each request in the decode queue, causing the user to wait for a complete answer for a long time, and reducing the user's interactive experience.
[0059] 2) Estimate the time consumption of prefill and decode according to experience, and prioritize the scheduling of the task with shorter time consumption.
[0060] The method can specifically include the following four steps: S210, receiving an inference request. S220, determining whether the estimated prefill time consumption is less than the decode time consumption. If yes, step S230 is performed; if no, step S240 is performed. S230, performing the prefill task. S240, performing the decode task. In step S220, a large number of repeated experiments are first performed manually, and then the experience value of the time consumption is obtained by summarizing a large number of experimental results, and the time consumption of the prefill task and the decode task is estimated based on the experience value.
[0061] However, the scheduling method relying on experience has limited accuracy. In the case of unchanged hardware configuration, the amount of tasks faced by the system is usually different at different times, and the fluctuation of the amount of tasks further causes the time for the device to perform the prefill or decode task to change, thereby deviating from the fixed experience value, which cannot guarantee that the hardware resources can be fully utilized at all times. Moreover, the experience value obtained through experiments does not have universality, and if the neural network model used by the system is replaced or the hardware configuration is changed, the existing experience value is no longer applicable, and new experiments need to be performed to summarize new experience values.
[0062] It can be seen that the above scheduling strategies cannot guarantee that the hardware resources can be efficiently utilized in real time, and cannot take into account the user's interactive experience while fully utilizing the hardware resources.
[0063] Therefore, the embodiments of the present application provide a scheduling method for inference tasks, which accurately evaluates the execution cost paid by the accelerator for performing each inference task, and then calculates the target inference task of the accelerator to be scheduled by the device according to the execution cost, thereby realizing the full utilization of the hardware resources of the accelerator, improving the inference efficiency, and taking into account the user's interactive experience.
[0064] In some possible embodiments, the method includes: the computing device determines the execution cost of the accelerator for performing each inference task, the execution cost being related to the execution progress of the inference task; wherein each inference task is a task performed by a different running stage in a neural network model; then, the computing device determines a target inference task from the inference tasks according to the execution cost, and then the computing device instructs the accelerator to perform the target inference task.
[0065] AsFigure 1 As shown, Figure 1 This is a schematic diagram of an exemplary application scenario provided by an embodiment of this application. Taking an intelligent question-answering system 100 as an example, the intelligent question-answering system 100 includes an inference task pool 110, a scheduling unit 120, and an accelerator cluster 130 (including accelerators 131-134).
[0066] The inference task pool 110 can be used to receive requests input by users to the system, and can also be used to store queues of inference tasks at different stages. The scheduling unit 120 is used to execute the inference task scheduling method provided in this application embodiment. The accelerator cluster 130 is used to run a neural network model to execute inference tasks according to the instructions of the scheduling unit 120.
[0067] For example, when a user inputs a prompt (request), the inference task pool 110 stores the request, and the scheduling unit 120 compares the execution cost of the inference task corresponding to the request with the execution cost of other inference tasks. The scheduling accelerator cluster 130 prioritizes the execution of the inference task with the lower execution cost.
[0068] Secondly, the system architecture of the embodiments of this application will be described by way of example.
[0069] like Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. The computing device 1000 includes at least: a memory 1010, a processor 1020, an accelerator 1030, and a bus 1040. The computing device 1000 may include one or more accelerators 1030. Figure 2 (Only one is shown in the image). In some feasible embodiments, computing device 1000 can be used to run... Figure 1 The intelligent question-and-answer system 100 shown.
[0070] The memory 1010 can be used to store the logic code corresponding to the scheduling method of the inference task provided in the embodiments of this application. Alternatively, the memory 1010 can store the logic code corresponding to the execution of a certain step by the computing device 1000 described in the following embodiments. In some feasible embodiments, the memory 1010 can be used to store... Figure 1 The data related to inference task pool 110 shown.
[0071] The processor 1020 can be used to determine the execution cost of each inference task performed by the accelerator, and to determine a target inference task from the inference tasks based on the execution cost, and instruct the accelerator to perform the target inference task. In some feasible embodiments, the processor 1020 can be used to run... Figure 1 The scheduling unit 120 shown.
[0072] Accelerator 1030 can be used to perform corresponding inference tasks according to the instructions of processor 1020 (such as receiving instructions sent by processor 1020). In some feasible embodiments, accelerator 1030 can be... Figure 1 One or more accelerators in the accelerator cluster 130 shown.
[0073] Optionally, the computing device 1000 can be a terminal device or a network device. The terminal device can include computers, mobile terminals, tablets, laptops, desktop computers, all-in-one computers, augmented reality (AR) devices, virtual reality (VR) devices, mixed reality (MR) devices, extended reality (ER) devices, personal digital assistants (PDAs), ultra-mobile personal computers (UMPCs), etc. The terminal device can also include edge computing devices (e.g., a box carrying a processing chip). The network device can include servers, etc.
[0074] Optionally, the memory 1010 may include random access memory (RAM), read-only memory (ROM), etc., wherein the RAM in the memory 1010 may run the necessary operating system, as well as modules such as the first determining module, the second determining module, and the indicating module for executing the scheduling method of the inference task provided in this application.
[0075] Optionally, the processor 1020 can be a central processing unit (CPU) or other general-purpose processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), digital signal processor (DSP) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0076] Alternatively, the accelerator 1030 can be built into the computing device 1000, with the corresponding connection method as follows: Figure 2As shown, the accelerator 1030 is connected to the processor 1020 via the bus 1040. In this case, the accelerator 1030 may specifically be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or an AI accelerator card, etc.
[0077] In some feasible embodiments, the accelerator 1030 may also be external to the computing device 1000 and communicate with the computing device 1000 via a universal serial bus (USB) such as USB-C, Ethernet, etc. In this case, the accelerator 1030 may specifically be one or more accelerators in a device or device cluster that include one or more accelerators (e.g., graphics cards) capable of performing inference tasks. For example, the accelerator 1030 may be one or more accelerators in a device such as a docking station containing graphics cards, an artificial intelligence (AI) server, a blade server, a high-density server, a rack server, or a distributed AI server cluster.
[0078] Optionally, the bus 1040 can be a peripheral component interconnect (PCI) bus or a high-speed peripheral component interconnect (PCIe) express bus, etc. For ease of illustration, Figure 2 The bus 1040 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1040 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1010, processor 1020).
[0079] It should be noted that the application scenarios and system architectures described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0080] For ease of understanding, the scheduling method for inference tasks provided in this application is described below with reference to the accompanying drawings. This scheduling method for inference tasks is applicable to... Figure 2 The computing device shown.
[0081] It should be noted that the execution of a certain step (such as S310 to S330 below) by the computing device described in the following embodiments can be understood as the processor 1020 executing the step.
[0082] Figure 3 A flowchart illustrating a scheduling method for an inference task according to an embodiment of this application is shown. The scheduling method for the inference task includes the following steps:
[0083] S310, The computing device determines the execution cost of the accelerator performing each inference task.
[0084] In the embodiments of this application, each inference task refers to a task executed at a different stage of the operation of the neural network model, such as the prefill and decode tasks in the inference process of the LLM mentioned above. The execution cost may be related to the execution progress of each inference task, or it may be related to the priority of each inference task, or the waiting time of each inference task, or the hardware parameters of the accelerator, or one or more of the feature parameters of the neural network model running on the accelerator.
[0085] The computing device may execute step S310 when it determines that the accelerator has completed a certain stage of the operation. Alternatively, the computing device may first confirm whether the accelerator meets the target state (such as whether it is idle or whether it has sufficient computing resources) when it receives a new request from the user. If it is determined that the accelerator meets the target state, it then executes step S310 to determine the execution cost of each inference task to be executed by the accelerator, and then determines the subsequent scheduling scheme of the accelerator. The computing device may also trigger the execution of step S310 under other circumstances, which is not limited in this application.
[0086] In step S310, the computing device determines the execution cost of each inference task performed by the accelerator. Specifically, this can mean determining the execution cost of each inference task performed by the accelerator based on the aforementioned parameters, or it can mean that the computing device obtains the execution cost of each inference task performed by the accelerator. In this possible implementation, the execution cost can be determined by other devices that can communicate with the computing device. For example, the execution cost can be determined by a server or edge computing device that can communicate with the computing device via the Internet, and then sent to the computing device.
[0087] It should be noted that the process of determining the execution cost will be described in detail in the following embodiments, and will not be repeated here.
[0088] For example, the CPU receives a task completion signal from the GPU, which indicates that the GPU has completed the prefill task for 10 requests (in the same batch) in the current request queue. Then, the CPU determines the execution cost of the GPU executing the prefill task to be 20 and the execution cost of the decode task to be 30 based on the execution progress of the current prefill task and decode task (e.g., the number of requests to be prefilled is 1, the number of requests in the decode queue is 8, etc.), the GPU's hardware parameters (e.g., the GPU can perform 5 billion floating-point operations per second, etc.) and the feature parameters of the neural network model running on the accelerator (e.g., the number of network layers, etc.).
[0089] S320, the computing device determines the target inference task from among the various inference tasks based on the execution cost.
[0090] In this step, the computing device can obtain the execution cost of each inference task performed by the accelerator through step S310. For example, the execution cost of the accelerator performing the prefill task is 20, and the execution cost of performing the decode task is 30. The computing device can determine the target inference task from each inference task using the following two possible implementation methods:
[0091] 1) The computing device determines the reasoning task corresponding to the lowest execution cost among all execution costs as the target reasoning task.
[0092] In this implementation, the computing device can sort the execution costs and then determine the inference task corresponding to the lowest execution cost as the target inference task.
[0093] For example, the CPU determines that the execution cost of the GPU performing the prefill task is 20 and the execution cost of performing the decode task is 30. The CPU sorts the two execution costs and, since the execution cost of the prefill task is the lowest, the CPU determines that the prefill task is the target inference task.
[0094] 2) The computing device determines the task with an execution cost less than or equal to a specified threshold as the target inference task.
[0095] In this implementation, the computing device can receive the execution cost threshold (specified threshold) set by the user. If all execution costs are greater than the specified threshold, it may indicate that the specified threshold is set incorrectly or that the current workload exceeds the load capacity of the computing device, etc. In this case, the computing device can display alarm information to prompt the user to modify the specified threshold or to detect related problems.
[0096] Furthermore, the specified threshold can also be related to the execution cost of each inference task. In other words, the computing device can determine the specified threshold based on the execution cost of each inference task, such as using the average or weighted average of the execution costs of each inference task as the specified threshold. This application does not impose any restrictions on this.
[0097] For example, the CPU receives an execution cost threshold (specified threshold) set by the user of 25, while the computing device determines that the execution cost of the aforementioned accelerator performing the prefill task is 20 and the execution cost of performing the decode task is 30. Since the execution cost of the prefill task is less than the specified threshold, the CPU determines that the prefill task is the target inference task.
[0098] For another example, the CPU takes the average of the execution cost of the prefill task (20) and the execution cost of the decode task (30) as a specified threshold, i.e., the specified threshold is 25. Since the execution cost of the prefill task is less than the specified threshold, the CPU determines that the prefill task is the target inference task.
[0099] S330, the computing device instructs the accelerator to perform the target inference task.
[0100] In one possible implementation, the computing device can generate target instructions and send them to the accelerator, which instruct the accelerator to perform a target inference task.
[0101] Specifically, the target instructions generated by the computing device may include necessary parameters such as the identifier of the accelerator and the identifier of the target inference task.
[0102] For example, the target instruction generated by the CPU includes the identifier "GPUA" of the accelerator and the identifier "prefill 01" of the target inference task. Then the CPU sends the target instruction to the accelerator "GPUA". After receiving the target instruction, the accelerator "GPUA" can first check whether the identifier is consistent with its own identifier. If the identifier is consistent, it executes the target inference task with the identifier "prefill 01".
[0103] Through the steps S310-S330 above, the computing device can achieve flexible and accurate inference task scheduling based on the execution cost of each inference task executed by the accelerator, making full use of the accelerator's computing power resources, improving inference efficiency, and enhancing user experience.
[0104] The process of determining the execution cost is described in detail below. In this embodiment, the execution cost can be related not only to the execution progress of each inference task, but also to the priority of each inference task, the waiting time of each inference task, the hardware parameters of the accelerator, or the feature parameters of the neural network model running on the accelerator. This embodiment provides at least four possible implementation methods for determining the execution cost of the accelerator executing each inference task. In this embodiment, any one of the following possible implementation methods can be selected for implementation, or a combination of implementation methods can be used:
[0105] 1) The computing device determines the execution time of each inference task performed by the accelerator, and then determines the execution cost of each inference task based on the execution time of each inference task.
[0106] In this possible implementation, the computing device determines the execution cost of each inference task by estimating the execution time of each inference task. Specifically, the computing device can determine the operators corresponding to each inference task executed by the accelerator and the execution order between the operators based on the feature parameters of the neural network model. Then, the computing device predicts the execution time of each operator based on the hardware parameters of the accelerator and the execution progress of each inference task. Based on the execution order and the execution time of each operator, the computing device determines the execution time of each inference task executed by the accelerator.
[0107] For example, the CPU determines the operators corresponding to the prefill and decode tasks executed by the GPU, as well as the execution order (including serial and parallel) of each operator, based on the feature parameters of the Large Language Model (LLM), such as the number of attention heads, the number of network layers, and the self-attention method. Then, based on the GPU's hardware parameters (e.g., the GPU can perform 5 billion floating-point operations per second), it predicts the execution time of each operator. The CPU then takes the maximum execution time among the parallel operators and sums it with the execution times of the other serially computed operators to finally determine the execution time of the GPU for the prefill and decode tasks. The formula for calculating the execution cost can be:
[0108] cost = duration
[0109] Where cost is the execution cost and duration is the execution time of the inference task. According to this calculation formula, the shorter the execution time of the inference task, the lower the execution cost. And the inference task with the lower execution cost is more likely to be scheduled for execution first, thus effectively improving the inference efficiency.
[0110] 2) The computing device determines the execution cost of each inference task by the accelerator based on the priority of each inference task.
[0111] For example, the formula for calculating the execution cost can be:
[0112]
[0113] Here, priority is the priority of the reasoning task. The priority can be a positive integer 1, 2, or 3. The larger the number, the higher the priority of the reasoning task. According to this calculation formula, the higher the priority of the reasoning task, the lower the execution cost. And the reasoning task with the lower execution cost is more likely to be scheduled and executed first. This can effectively avoid high-priority tasks from waiting for a long time.
[0114] 3) The computing device determines the execution cost of the accelerator executing each inference task based on the waiting time of each inference task.
[0115] For example, the formula for calculating the execution cost can be:
[0116]
[0117] Among them, time wait Let be the waiting time of the inference task. According to this calculation formula, the longer the waiting time of the inference task, the lower the execution cost. The lower the execution cost of the inference task, the higher the probability that it will be scheduled for execution first. This can realize that the execution cost of the inference task decreases dynamically during the waiting process, rather than remaining fixed. Accordingly, the probability of being scheduled for execution first increases, thus effectively avoiding long waiting times for inference tasks.
[0118] 4) The computing device determines the execution cost of each inference task performed by the accelerator based on the execution cost calculation model.
[0119] The execution cost calculation model is a mathematical model constructed based on the hardware parameters of the accelerator and the feature parameters of the neural network model running on the accelerator.
[0120] Furthermore, the computing device can input the execution progress of each inference task into the execution cost calculation model to obtain the execution cost of the accelerator executing each inference task.
[0121] In the embodiments of this application, in addition to selecting any one of the above possible implementation methods, a combination of implementations can also be used. For example, the computing device can determine the execution cost of each inference task based on the execution time, priority, and waiting time of each inference task.
[0122] For example, the formula for calculating the execution cost can be:
[0123]
[0124] The formula involves three parameters: execution time of the inference task, priority, and waiting time. This allows for a more comprehensive and accurate assessment of the execution cost of each inference task, further improving the accuracy of scheduling.
[0125] In this embodiment, the neural network model running on the accelerator is one of several models, and the corresponding inference task includes multiple tasks at different stages. For example, the neural network model running on the accelerator can be a Large Language Model (LLM), with corresponding inference tasks including prefill and decode tasks; the neural network model running on the accelerator can also be a Generative Adversarial Network (GAN) model, with corresponding inference tasks including generator tasks and discriminator tasks; the neural network model running on the accelerator can also be other models with similar encoder and decoder structures, with corresponding inference tasks including encoding and decoding tasks.
[0126] The following describes the scheduling method for inference tasks provided in this application, taking the neural network model running on the accelerator as the Large Language Model (LLM) and the corresponding inference tasks including prefill and decode tasks.
[0127] like Figure 4 As shown, Figure 4 This application provides a flowchart illustrating a scheduling method for inference tasks, which may involve one or more modules, such as... Figure 4 The module includes an operator flowchart construction module, an execution time construction module, a load awareness module, an execution cost calculation module, and a scheduling decision module. These modules can be included in the above. Figure 2 The computing device 1000 shown may also be included in other devices that can communicate with the computing device 1000.
[0128] S410, input the feature parameters of the neural network model.
[0129] In this step, specifically, the feature parameters of the neural network model can be input into... Figure 4 The operator flowchart construction module in the system is used to construct the operator flowchart of the neural network model based on the feature parameters of the neural network model. The operator flowchart can include the input shape, output shape, computational load, input data volume, output data volume, and communication data volume of the operators involved in the inference task.
[0130] To further improve the accuracy of operator flowchart construction, parallel strategy parameters related to the neural network model (such as tensor parallelism / pipeline parallelism) and other optimization strategies can be input into the operator flowchart construction module.
[0131] For example, taking feature parameters from the large language model parameters such as hidden layer dimension, number of attention heads, number of layers, and self-attention method, as well as runtime batch size, input and output sequence lengths, tensor parallelism, pipeline parallelism, data parallelism, and expert parallelism strategies as input, the operator flowchart construction module generates operator flowcharts for the prefill and decode tasks respectively. For instance, Figure 5 The operator flowchart shown is as follows, in which, Figure 5 This is a schematic diagram of an operator flowchart provided in an embodiment of this application, which includes four operators, namely operator 1 to operator 4. During inference, the model first executes operator 1 on the input data, then executes operator 2 and operator 3 in parallel, and finally executes operator 4 to obtain the output data.
[0132] S420, input operator flowchart and accelerator hardware parameters.
[0133] In this step, specifically, the operator flowchart obtained in step S410 and the acquired accelerator hardware parameters can be input into... Figure 4 The module for constructing execution time in the program will build execution time calculation models for each operator based on these two parameters, as well as execution time calculation models for the prefill and decode tasks respectively.
[0134] For example, the operator-level flowcharts corresponding to the prefill and decode tasks, along with parameters such as the accelerator's computing power, memory access bandwidth, network topology, and communication algorithms, are input into the execution time construction module. The calculation formula for the execution time of each operator can be: Execution Time = Computation Time + Memory Access Time + Communication Time + Waiting Time. Furthermore, the execution time construction module can also consider the characteristics of the accelerator's configured computing mode when constructing the execution time calculation model. For instance, if the accelerator is in roofline computing mode, since multiple computing processes are performed in parallel in this mode, the corresponding execution time calculation model for a single operator can be: Execution Time = max(Computation Time, Memory Access Time, Communication Time) + Waiting Time. Therefore, the execution time calculation model can be based on the aforementioned... Figure 5 The execution order of the various operators shown determines the execution time calculation model for the prefill and decode tasks, respectively. Figure 5 For example, if Figure 5The operator shown corresponds to the decode task. The execution time calculation model for the decode task can be: execution time of operator 1 + max(execution time of operator 2, execution time of operator 3) + execution time of operator 4. In this way, the execution time construction module can be combined with the aforementioned execution time calculation models of each operator to obtain the specific execution time calculation model of the prefill or decode task.
[0135] S430, Input execution time calculation model.
[0136] In this step, specifically, the execution time calculation model obtained in step S420 can be input into... Figure 4 The execution cost calculation module in the middle can then construct an execution cost calculation model for the accelerator to execute various inference tasks (such as prefill and decode tasks) based on the input execution cost calculation module.
[0137] In one possible implementation, the execution cost calculation model can also be related to the priority of each inference task or the waiting time of each inference task.
[0138] For example, the execution cost calculation module is constructed based on the execution time calculation model. The execution cost calculation model for the decode task can be:
[0139] cost = duration
[0140] Where duration is the execution time required for the accelerator to decode a new token for all requests in the decode queue; and the execution cost calculation model for the prefill task can be:
[0141]
[0142] Where duration is the execution time of the accelerator in completing the prefill tasks for all requests in the prefill queue, and priority is the priority of all requests in the prefill queue, which can be the average priority of all requests. wait This represents the waiting time for all requests in the prefill queue, specifically the average waiting time for all requests.
[0143] S440, Input the execution progress of each inference task.
[0144] It should be noted that steps S410-S430 related to constructing the execution time calculation model can be executed only once if the hardware parameters of the accelerator and the feature parameters of the neural network model remain constant. However, if any of the aforementioned parameters change, steps S410-S430 can be executed again to construct a new execution time calculation model, which is then input into the execution cost calculation module to construct a new execution cost calculation model. Therefore, the inference task scheduling method in this embodiment can not only adapt to various neural network models and accelerators, but also construct an accurate execution cost calculation model in real time based on changes in the neural network model or accelerator during implementation, so as to accurately calculate the execution cost of each inference task executed by the accelerator. Since the execution progress changes in real time, steps S440-S480 can be executed cyclically to achieve optimal real-time scheduling.
[0145] In this step, specifically, the execution progress of the prefill or decode task can be input into... Figure 4 The load awareness module in the system includes one or more parameters in the execution progress of the prefill task, such as the length of the prefill queue, the priority and waiting time of each request in the prefill queue, and the length of the prompt word; the execution progress of the decode task includes one or more parameters in the number of requests to be decoded, the current decoded step size, and the time. The load awareness module can then measure the current system load based on these parameters.
[0146] S450, Input load status.
[0147] In this step, specifically it can be Figure 4 The load awareness module inputs the load status into the execution cost calculation module.
[0148] In one possible implementation, the load-aware module can input the acquired multiple parameters as a parameter set into the execution cost calculation module.
[0149] For example, the load awareness module obtains a prefill queue with a length of 1, a priority of 5, and a waiting time of 60 seconds, writes it into a specified structure {1, 5, 60}, and inputs it into the execution cost calculation model.
[0150] S460, Input execution cost.
[0151] In this step, specifically it can be Figure 4 The execution cost calculation module in the system inputs the calculated execution cost required for the accelerator to execute each inference task into the scheduling decision module.
[0152] For example, the execution cost calculation module evaluates the execution cost of the accelerator executing the prefill task as 5 and the execution cost of the accelerator executing the decode task as 10 based on the execution cost calculation model constructed from the time consumption calculation model, and inputs these values into the scheduling decision module.
[0153] In one possible implementation, the execution cost calculation module can inversely solve for parameters used to measure the execution progress of each inference task based on a specified execution cost threshold. This allows for direct comparison with the inversely calculated parameters after obtaining the execution progress, thereby improving the efficiency of scheduling decisions.
[0154] S470, Input scheduling scheme.
[0155] In this step, specifically it can be Figure 4 The scheduling decision module inputs the scheduling plan into the scheduling execution module. In this step, the scheduling decision module can first determine the scheduling plan based on the execution cost of each inference task executed by the accelerator.
[0156] For example, if the execution cost obtained by the scheduling decision module is 5 for the accelerator to execute the prefill task and 10 for the accelerator to execute the decode task, then the scheduling decision module can generate the next scheduling plan according to the principle of minimizing the execution cost, that is, execute the prefill task, and input the scheduling plan into the scheduling execution module.
[0157] S480, execute the scheduling plan.
[0158] In this step, specifically it can be Figure 4 The scheduling execution module instructs the accelerator to execute the scheduling scheme input in step S470, for example, generating target instructions for instructing the accelerator to execute the target inference task in the scheduling scheme and sending them to the accelerator.
[0159] Through steps S410-S480 above, the collaborative computation of each exemplary module in the computing device enables the construction of an execution cost calculation model for each inference task performed by the accelerator, based on the hardware parameters of the current accelerator and the feature parameters of the neural network model. By inputting the execution progress of each inference task into this model, the execution cost required by the accelerator to perform each inference task can be accurately predicted. Then, based on the execution cost, the accelerator can be flexibly and accurately scheduled to prioritize the execution of inference tasks with lower execution costs, making full use of the accelerator's hardware resources, improving inference efficiency, avoiding long waiting times for users to complete inference results, improving system interactivity, and enhancing user experience.
[0160] It should be noted that the above Figure 4The division of the various modules is only an exemplary method. In some feasible embodiments, it can also be divided into more or fewer modules. For example, it can be divided into... Figure 4 The functions of the operator flowchart construction module, execution time construction module, and load awareness module (highlighted by the dashed box) are all contained within a more complex execution cost calculation module, such as... Figure 6 As shown, Figure 6 This is a flowchart illustrating another inference task scheduling method provided in this application embodiment. The feature parameters and parallel strategy (or corresponding operator flowchart) of the neural network model, hardware parameters, and the execution progress of each inference task are all input to... Figure 6 In the execution cost calculation module, the execution cost of each inference task executed by the accelerator is directly output by the execution cost calculation module. The subsequent process can refer to the above steps S460-S480, which will not be repeated here.
[0161] The foregoing mainly describes the solutions of the embodiments of this application from a methodological perspective. It is understood that, in order to achieve the above functions, the scheduling device for inference tasks includes at least one of the hardware structures and software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.
[0162] This application embodiment can divide the scheduling device for inference tasks into functional units based on the above method example. For example, each function can be divided into its own functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0163] For example, Figure 7 This is a schematic diagram of the structure of a scheduling device 800 for inference tasks provided in an embodiment of this application. The scheduling device 800 for inference tasks is applied in a computing device, or the scheduling device 800 for inference tasks can be a computing device itself. The scheduling device 800 for inference tasks includes:
[0164] The first determining module 810 is used to determine the execution cost of each inference task performed by the accelerator, wherein the execution cost is related to the execution progress of the inference task; wherein each inference task is a task performed in a different running stage of the neural network model.
[0165] The second determining module 820 is used to determine the target reasoning task from the various reasoning tasks based on the execution cost.
[0166] Instruction module 830 is used to instruct the accelerator to perform the target inference task.
[0167] For example, combining Figure 3 The first determining module 810 can be used to perform, for example... Figure 3 As shown in S310, the second determining module 820 can be used to perform, for example... Figure 3 As shown in S320, the indicator module 830 can be used to perform, for example... Figure 3 The S330 shown.
[0168] In one possible implementation, the execution cost is related to the priority of each of the inference tasks, or the waiting time of each of the inference tasks, or the hardware parameters of the accelerator, or the feature parameters of the neural network model running on the accelerator.
[0169] In one possible implementation, the first determining module 810 is further configured to determine the execution time of the accelerator in performing each inference task; and determine the execution cost based on the execution time of each inference task.
[0170] In one possible implementation, the first determining module 810 is further configured to: determine, based on the feature parameters of the neural network model, the operators corresponding to each inference task executed by the accelerator and the execution order among the operators; predict the execution time of each operator based on the hardware parameters of the accelerator and the execution progress of each inference task; and determine the execution time of the accelerator executing each inference task based on the execution order and the execution time of each operator.
[0171] In one possible implementation, the first determining module 810 is further configured to determine the execution cost of each inference task performed by the accelerator based on an execution cost calculation model, wherein the execution cost calculation model is a mathematical model constructed based on the hardware parameters of the accelerator and the feature parameters of the neural network model running on the accelerator.
[0172] In one possible implementation, the first determining module 810 is further configured to input the execution progress of each inference task into the execution cost calculation model to obtain the execution cost of the accelerator executing each inference task.
[0173] In one possible implementation, the first determining module 810 is further configured to determine the execution cost of the accelerator executing each inference task based on the priority of each inference task.
[0174] In one possible implementation, the first determining module 810 is further configured to determine the execution cost of the accelerator executing each inference task based on the waiting time of each inference task.
[0175] In one possible implementation, the neural network model is a Large Language Model (LLM), and the inference task includes a prefill task and a decode task.
[0176] As a feasible example, the inference task scheduling device 800 provided in this application is implemented through a software module. For example, the software module can be provided to users through a cloud service subscription model, and users can choose different subscription levels according to their needs. Alternatively, the software module can also provide enterprise-level customized services with professional domain customization, interface personalization and extended functions according to the needs of users or enterprises.
[0177] Furthermore, the inference task scheduling device 800 provided in this application can also be provided to users as a value-added service, and this application does not limit this. When the inference task scheduling device 800 is implemented through a software module, the inference task scheduling device 800 can be embedded into other model inference service systems, or it can be sold separately as inference service scheduling software.
[0178] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to perform a scheduling method for an inference task.
[0179] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a scheduling method for an inference task, or instruct the computing device to perform a scheduling method for an inference task.
[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.
Claims
1. A scheduling method for inference tasks, characterized in that, The method comprises: determining an execution cost of each inference task executed by the accelerator, the execution cost being related to an execution progress of the inference task; wherein each inference task is a task executed by a different running stage in the neural network model; determining a target inference task from the inference tasks according to the execution cost; instructing the accelerator to execute the target inference task.
2. The method of claim 1, wherein, The execution cost is related to a priority of each inference task, or a waiting time of each inference task, or a hardware parameter of the accelerator, or a characteristic parameter of a neural network model running on the accelerator.
3. The method according to claim 1 or 2, characterized in that, The determining of the execution cost of each inference task executed by the accelerator comprises: determining an execution time of each inference task executed by the accelerator; determining the execution cost according to the execution time of each inference task.
4. The method of claim 3, wherein, The determining of the execution time of each inference task executed by the accelerator comprises: determining an operator corresponding to each inference task executed by the accelerator and an execution order between the operators according to a characteristic parameter of the neural network model; predicting an execution time of each operator according to a hardware parameter of the accelerator and an execution progress of each inference task; determining the execution time of each inference task executed by the accelerator according to the execution order and the execution time of each operator.
5. The method according to any one of claims 1 to 4, characterized in that, The determining of the execution cost of each inference task executed by the accelerator comprises: determining the execution cost of each inference task executed by the accelerator based on an execution cost calculation model, the execution cost calculation model being a mathematical model constructed according to a hardware parameter of the accelerator and a characteristic parameter of a neural network model running on the accelerator.
6. The method of claim 5, wherein, The determining of the execution cost of each inference task executed by the accelerator based on the execution cost calculation model comprises: inputting the execution progress of each inference task into the execution cost calculation model to obtain the execution cost of each inference task executed by the accelerator.
7. The method according to any one of claims 1 to 6, characterized in that, The determining of the execution cost of each inference task executed by the accelerator comprises: determining the execution cost of each inference task executed by the accelerator according to a priority of each inference task.
8. The method according to any one of claims 1 to 7, characterized in that, The determining of the execution cost of each inference task executed by the accelerator comprises: determining the execution cost of each inference task executed by the accelerator according to a waiting time of each inference task.
9. The method according to any one of claims 1 to 8, characterized in that, The neural network model is a large language model (LLM), and the inference task comprises a prefill task and a decode task.
10. A scheduling device for reasoning tasks, characterized in that, The device comprises: a first determining module configured to determine an execution cost of each inference task executed by the accelerator, the execution cost being related to an execution progress of the inference task; wherein each inference task is a task executed by a different running stage in the neural network model; a second determining module configured to determine a target inference task from the inference tasks according to the execution cost; an instructing module configured to instruct the accelerator to execute the target inference task.
11. The apparatus of claim 10, wherein, The execution cost is related to a priority of each inference task, or a waiting time of each inference task, or a hardware parameter of the accelerator, or a characteristic parameter of a neural network model running on the accelerator.
12. The apparatus of claim 10 or 11, wherein, The first determining module is further configured to: determine execution time of the accelerator in executing the inference tasks; determine the execution cost according to the execution time of the inference tasks.
13. The apparatus of claim 12, wherein, The first determining module is further configured to: determine operators corresponding to the inference tasks and execution order between the operators according to the feature parameters of the neural network model; predict execution time of the operators according to the hardware parameters of the accelerator and execution progress of the inference tasks; determine execution time of the accelerator in executing the inference tasks according to the execution order and the execution time of the operators.
14. The apparatus of any one of claims 10-13, wherein, The first determining module is further configured to: determine execution cost of the accelerator in executing the inference tasks based on an execution cost calculation model, the execution cost calculation model being a mathematical model constructed according to the hardware parameters of the accelerator and feature parameters of the neural network model running on the accelerator.
15. The apparatus of claim 14, wherein, The first determining module is further configured to: input the execution progress of the inference tasks into the execution cost calculation model to obtain the execution cost of the accelerator in executing the inference tasks.
16. The apparatus of any one of claims 10-15, wherein, The first determining module is further configured to: determine execution cost of the accelerator in executing the inference tasks according to the priority of the inference tasks.
17. The apparatus of any of claims 10-16, wherein, The first determining module is further configured to: determine execution cost of the accelerator in executing the inference tasks according to the waiting time of the inference tasks.
18. The apparatus of any of claims 10-17, wherein, The neural network model is a large language model (LLM), and the inference tasks include prefill (prefill) tasks and decoding (decode) tasks.
19. A computing device, comprising: The computing device includes a processor and a memory; the processor is coupled with the memory; the memory is used to store computer instructions, the computer instructions are loaded and executed by the processor to enable the computing device to implement the inference task scheduling method as claimed in any one of claims 1 to 9.
20. A computer program product, characterised in that, When the computer program product is running in the computing device, the computing device executes the inference task scheduling method as claimed in any one of claims 1 to 9.