Heterogeneous reasoning computing power scheduling method and device and storage medium

Abstract

The invention provides a heterogeneous reasoning computing power scheduling method and device and a storage medium, and the method comprises the steps: receiving a reasoning request, splitting the reasoning request, and obtaining at least one reasoning task; according to the coding length of each reasoning task, scheduling the reasoning tasks to a long task queue or a short task queue; when it is monitored that reasoning tasks with to-be-processed task states exist in the long task queue and the short task queue, the working state of the reasoning service of the first processor is judged based on the number of tasks in processing of the first processor in the long task queue and the short task queue and the maximum concurrency of the first processor; the maximum concurrency of the first processor is obtained based on the current conditions of the long task queue and the short task queue; and on the basis of the working state of the first processor reasoning service, the reasoning task to be processed is scheduled to the first processor reasoning service or the second processor reasoning service so as to maximize and reasonably utilize the mixed computing power resource of the heterogeneous processor.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of large model inference deployment, in particular to a heterogeneous inference computing power scheduling method and device and storage medium. BACKGROUND

[0002] In the scheduling problem of AI inference large model inference computing power, the graphics processing unit (GPU) has become the mainstream inference platform due to its parallel computing advantage. The larger the inference model parameter quantity and the more the number of concurrent inference requests, the more GPU computing power resources are needed. However, GPU resources are expensive, and constantly increasing GPU resources may bring huge cost investment. The central processing unit (CPU) deployed on the AI inference server is often in an idle state, resulting in waste of computing resources of the AI inference server. CPU and GPU hybrid heterogeneous computing power inference can be used to solve the problem of idle CPU resources. However, the current CPU and GPU hybrid heterogeneous computing power inference cannot effectively maximize the rational use of GPU and CPU hybrid computing power resources. SUMMARY

[0003] The present disclosure provides a heterogeneous inference computing power scheduling method, device and system, which can maximize the rational use of hybrid computing power resources of heterogeneous processors.

[0004] To solve the above technical problems, the present disclosure provides a heterogeneous inference computing power scheduling method, which is applied to a heterogeneous computing power scheduling system, the system comprising a first processor inference service and a second processor inference service, the system being configured with two task queues at startup, the two task queues comprising a long task queue and a short task queue, the method comprising: receiving an inference request, splitting the inference request to obtain at least one inference task, and scheduling the inference tasks to the long task queue or the short task queue according to the encoding length of each inference task; when it is monitored that there is an inference task to be processed in the two task queues, judging the working state of the first processor inference service based on a first task number and a first processor maximum concurrency, the first processor maximum concurrency being obtained based on the current conditions of the two task queues, and the first task number being the number of tasks in the two task queues being processed by the first processor inference service; based on the working state of the first processor inference service, scheduling the inference task to be processed to the first processor inference service or the second processor inference service.

[0005] The embodiment of the present disclosure further provides a heterogeneous inference computing power scheduling device, the device is configured in a heterogeneous computing power scheduling system, the system includes a first processor inference service and a second processor inference service, the system is configured with two task queues when starting, the two task queues include a long task queue and a short task queue, the device includes: A task division module is configured to receive an inference request, split the inference request to obtain at least one inference task, and schedule the inference tasks to the long task queue or the short task queue according to the encoding length of each inference task. A state acquisition module is configured to judge the working state of the first processor inference service based on the first task number and the first processor maximum concurrency when monitoring that there is an inference task to be processed in the two task queues, the first processor maximum concurrency is obtained based on the current condition of the two task queues, and the first task number is the number of tasks in the two task queues processed by the first processor inference service. A task scheduling module is configured to schedule the inference task to be processed to the first processor inference service or the second processor inference service based on the working state of the first processor inference service.

[0006] The embodiment of the present disclosure further provides an electronic device, including: A memory is configured to store a computer program. A processor is configured to execute the computer program to implement the steps of any one of the heterogeneous inference computing power scheduling methods provided by the embodiments of the present disclosure.

[0007] The embodiment of the present disclosure further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the steps of any one of the heterogeneous inference computing power scheduling methods provided by the embodiments of the present disclosure.

[0008] The embodiment of the present disclosure further provides a computer program product, including a computer program, and the computer program is executed by a processor to implement the steps of any one of the heterogeneous inference computing power scheduling methods provided by the embodiments of the present disclosure.

[0009] The disclosure provides a heterogeneous inference computing power scheduling method. By splitting inference requests into different long and short inference task types, the affinity mechanism of long and short inference task types and different processors is realized, the inference tasks of different processor affinities are preferentially scheduled to the inference service of the corresponding processor, and the difference between the inference performance of different processors is reduced. By introducing long / short task queues, different types of inference tasks are more reasonably scheduled to the inference service of different processors, the mixed computing power of heterogeneous processors is maximized, and the overall throughput of the inference service is improved. At the same time, it is realized without monitoring the utilization rate and the concurrency of the processor inference service. While scheduling the inference task, the concurrency of the processor inference service at the current time can be synchronously obtained from the task queue, avoiding the problem of incorrect scheduling caused by the asynchronization of performance monitoring and scheduling decision time. According to the current long / short task queue, the maximum concurrency of the processor under the current inference task condition is dynamically calculated, solving the problem that the fixed concurrency cannot cope with various concurrent inference tasks, and maximizing the concurrency of the processor, thereby maximizing the inference performance. The dynamic scheduling process is realized, solving the problem that the fixed proportion allocation method cannot adapt to the dynamic load change of the inference scene. The mixed computing power resources of the heterogeneous processor are fully utilized, solving the problem that the first processor of the AI inference server runs at high load, while the second processor resource is idle and wasted.

[0010] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the disclosure, nor is it used to limit the scope of the disclosure. Other features of the disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are used to better understand the scheme and do not limit the disclosure. Among them: Figure 1 A flowchart of a heterogeneous inference computing power scheduling method provided by an embodiment of the disclosure; Figure 2 A flowchart of another heterogeneous inference computing power scheduling method provided by an embodiment of the disclosure; Figure 3 A scheduling flowchart of a heterogeneous scheduling service provided by an embodiment of the disclosure; Figure 4 A flowchart of another heterogeneous inference computing power scheduling method provided by an embodiment of the disclosure; Figure 5 A flowchart of another heterogeneous inference computing power scheduling method provided by an embodiment of the disclosure; Figure 6 A flowchart of another heterogeneous inference computing power scheduling method provided by an embodiment of the disclosure; Figure 7A batch encoder processing flowchart provided by an embodiment of the present disclosure; Figure 8 A flowchart of a reasoning performance test of a scheduler provided by an embodiment of the present disclosure; Figure 9 A processing flowchart of a reasoning task scheduling of a scheduler provided by an embodiment of the present disclosure; Figure 10 A batch decoder processing flowchart provided by an embodiment of the present disclosure; Figure 11 A structure schematic diagram of still another heterogeneous reasoning computing power scheduling device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Thus, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0013] In the related art, the current CPU+GPU mixed heterogeneous computing power inference is in the development stage. The current mainstream scheduling strategy is to divide according to a fixed ratio or to schedule according to the detection of GPU / CPU resource utilization, for example, 80% of the inference requests are scheduled to the GPU, and 20% are scheduled to the CPU, or when the GPU is idle, the inference requests are scheduled to the GPU, and when the GPU is fully loaded, the inference requests are scheduled to the CPU. However, the actual inference scene is often complex, the concurrency of the request, the length of the inference prompt, and the CPU / GPU computing state are all changing dynamically. The static ratio division method cannot dynamically schedule the inference task, and cannot adapt to the dynamic load change of the inference scene. The scheduling method based on the detection of GPU / CPU resource utilization cannot realize real-time monitoring and concurrent control in the true sense, and may cause the detected resource utilization to be not the real resource utilization at the current scheduling time, resulting in incorrect scheduling of the inference task. Therefore, the GPU+CPU computing power is not maximally utilized, and the inference performance may be reduced. This is because the hardware utilization detection and the scheduling decision time are not synchronized. The hardware utilization detection has a periodic detection feature, and the hardware utilization data obtained at the scheduling decision time is often not the real data at the current time. This delay may cause incorrect scheduling judgment, and may cause the GPU inference service to have idle resources, but because the hardware detection was not in the idle state last time, the inference task is incorrectly scheduled to the CPU inference service, so that the overall inference performance is lower than that of the pure GPU inference service. In addition, the concurrent data acquisition and the scheduling decision time are not synchronized. The concurrent data acquisition of the inference service has periodicity and delay characteristics. The concurrent number of the inference service obtained at the scheduling decision time is often not the real concurrent number at the current time. This delay may cause incorrect scheduling judgment, and may cause the concurrent tasks scheduled to the GPU / CPU to not match the current idle concurrent number of the GPU / CPU, which may cause a large number of requests to be blocked or the concurrent number to not reach the maximum concurrent number, and the GPU and CPU mixed computing power cannot be effectively maximized.

[0014] Therefore, the current CPU and GPU mixed heterogeneous computing power inference cannot maximize the overall computing power resources of the server while meeting the service level objective (SLO) indicators.

[0015] Therefore, the current CPU and GPU mixed heterogeneous computing power inference cannot maximize the overall computing power resources of the server while meeting the service level objective (SLO) indicators.

[0016] The heterogeneous inference computing power scheduling method, device and storage medium of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0017] Figure 1is a flowchart of a heterogeneous inference computing power scheduling method according to an embodiment of the disclosure, as shown in the figure, the method can include the following steps: Figure 1 Step 101, receiving an inference request, splitting the inference request to obtain at least one inference task; and scheduling the inference tasks to a long task queue or a short task queue according to the encoding length of each inference task.

[0018] The heterogeneous inference computing power scheduling method of the embodiments of the present application is applied to a heterogeneous computing power scheduling system, which includes a first processor inference service (such as a GPU inference service) and a second processor inference service (such as a CPU inference service). The system is configured with two task queues when starting, including a long task queue and a short task queue.

[0019] In one example, as shown in the figure, Figure 2 The mixed heterogeneous computing power scheduling system in a computing resource limited environment includes a heterogeneous scheduling service, a GPU inference service and a CPU inference service. The heterogeneous scheduling service is located in the middle layer above the GPU inference service and the CPU inference service, used to receive an inference request sent by a user, and select to schedule the inference request to the GPU inference service or the CPU inference service according to the calculation result of the scheduling decision algorithm; the GPU inference service is used to call a large model inference service of GPU computing power, receive the inference request scheduled by the heterogeneous scheduling service, and call the GPU computing power to calculate the inference result; the CPU inference service is used to call a large model inference service of CPU computing power, receive the inference request scheduled by the heterogeneous scheduling service, and call the CPU computing power to calculate the inference result.

[0020] In this embodiment, the heterogeneous scheduling service receives an inference request, splits the inference request to obtain at least one inference task, and schedules the inference tasks to a long task queue or a short task queue according to the encoding length of each inference task.

[0021] In some embodiments, as shown in the figure, Figure 3 The heterogeneous computing power scheduling system is configured with a batch encoder, a task buffer, a scheduler, a hardware performance fingerprint library and a batch decoder; the heterogeneous scheduling service receives an inference request sent by a user, the inference request first enters the batch encoder for preprocessing, the processed inference request is scheduled to the task buffer, and the scheduler schedules the inference tasks in the task buffer to the GPU inference service or the CPU inference service according to the inference performance data stored in the hardware performance fingerprint library.

[0022] In some embodiments, the above method of splitting the inference request to obtain at least one inference task, and scheduling the inference tasks to a long task queue or a short task queue according to the encoding length of each inference task, as shown in the figure, Figure 4 includes the following steps:​ Step 1011, according to the prompt word included in the reasoning request, the reasoning request is divided into at least one reasoning task, and the at least one reasoning task is bound with the unique request identification of the reasoning request, the number of the at least one reasoning task is the same as the number of the prompt words included in the reasoning request.

[0023] It should be noted that a single reasoning request may include one prompt or multiple prompts, for example, the prompts of the reasoning request are "introduce xxx long prompt", "introduce xxx short prompt", "introduce CPU", "hello world!", and "ni hao!".

[0024] In some embodiments, when a single reasoning request includes multiple prompts, the batch encoder splits the multiple prompts of the single reasoning request into multiple reasoning tasks, and binds the multiple reasoning tasks with the request ID of the reasoning request. When a single reasoning request includes only one prompt, no splitting is performed, and it is converted into one reasoning task.

[0025] Step 1012, encode each reasoning task in the at least one reasoning task into a wordpiece sequence, and obtain the sequence length of the wordpiece sequence.

[0026] In some embodiments, the batch encoder uses the tokenizer of the reasoning model to perform encoding processing on each reasoning task (i.e., prompt), converts the text into a wordpiece sequence, and then calculates the length of the wordpiece sequence.

[0027] Step 1013, scheduling the reasoning task with a sequence length greater than the long-short classification threshold to the long task queue, and scheduling the reasoning task with a sequence length less than or equal to the long-short classification threshold to the short task queue; wherein the long-short classification threshold is obtained by the system based on the user service level objective indicators and the second reasoning performance data of the second processor reasoning service in the initialization stage.

[0028] In some embodiments, in the initialization stage of the heterogeneous computing power scheduling system, the long-short classification threshold is calculated based on the user service level objective indicators and the reasoning performance data of the second processor reasoning service and stored.

[0029] In some embodiments, the user service level objective indicators (SLO indicators) include the maximum delay of the first wordpiece (seconds) and the maximum interval between generating each wordpiece (seconds), and the SLO indicator data is specified by the user when the reasoning system is started.

[0030] In some embodiments, the method further comprises: obtaining first processor information corresponding to the first processor inference service and second processor information corresponding to the second processor inference service; determining whether the first inference performance data corresponding to the first processor inference service and the second inference performance data corresponding to the second processor inference service are stored in the first storage structure based on the first processor information and the second processor information; if not, configuring the IP and port of the second processor inference service and the IP and port of the first processor inference service; calling the OpenAI API interface of the second processor inference service and the first processor inference service to perform inference performance testing, and obtaining the first inference performance data and the second inference performance data; and storing the first inference performance data and the second inference performance data in the first storage structure.

[0031] In some embodiments, the first storage structure is a hardware performance fingerprint library that records inference performance data of different CPU models (such as Intel Xeon vs. AMD EPYC) and GPU models (such as NVIDIA A100 vs. V100). The inference performance data is obtained from two parts: one part is built-in data, which is pre-tested by inference system developers and imported; the other part is inference performance data obtained after triggering the CPU / GPU performance test process when the GPU / CPU model of the current inference system is not in the built-in data.

[0032] In some embodiments, the CPU / GPU performance test process is as shown in Figure 4 First, the IP and port of the CPU inference service and the IP and port of the GPU inference service are configured, then a streaming request is constructed, and the OpenAI API interface of the CPU inference service and the GPU inference service is called respectively to perform inference performance testing. The first token arrival time TTFT (Time to First Token) and the subsequent token arrival time are recorded in the test code, the average time interval TPOT (Time Per Output Token) is calculated, and after the test is completed, the prefill performance and decode performance are calculated based on the input data, the return data and the recorded time points.

[0033] In some embodiments, the inference performance data includes prefill performance and decode performance, and the prefill performance includes the number of concurrents corresponding to different input lengths, and the number of concurrents corresponds to the first token delay (TTFT) and the throughput. In one example, the storage structure of the number of concurrents corresponding to different input lengths of a certain type of CPU is: input length -> {concurrent number: (TTFT, throughput)…}, such as: 128: {1: (0.08, 1600), 4: (0.12, 1067), …}, 512: {1: (0.25, 2048), 4: (0.38, 1347), …}, 1024: {1: (0.48, 2133), 4: (0.72, 1422), …} If is 0.26, the maximum input length that meets the requirement of 0.26 among the above two data is 512.

[0034] In an example, the prefill performance The calculation formula is as follows:

[0035] wherein, is the number of input tokens, and TTFT is the first token time delay.

[0036] In an example, the decode performance The calculation formula is as follows:

[0037] wherein, is the average time interval, and the performance data unit is tokens / s.

[0038] In some embodiments, the method for obtaining the long-short classification threshold value comprises: obtaining the maximum input length that meets the first token maximum delay in the second inference performance data, and taking the maximum input length as the long-short classification threshold value.

[0039] In an example, the long-short classification threshold value is calculated by the following formula:

[0040] wherein, represents the maximum input length that meets the condition in the inference performance data corresponding to the CPU inference service.

[0041] Since the inference performance data corresponding to the CPU inference service is stored in the hardware performance fingerprint library, the long-short classification threshold value can also be represented as the maximum input length that meets the condition in the inference performance data corresponding to the CPU inference service in the hardware performance fingerprint library. It can be understood that the inference performance data of the CPU inference service corresponding to different models of CPUs is different, and the inference performance data of the CPU inference service corresponding to each model of CPU includes multiple input lengths, and the maximum input length that meets the one or more primary input lengths under the condition, and the maximum input length is obtained from the primary input lengths.

[0042] The embodiment can reduce the initialization data detection time of the scheduling system by storing the inference performance data in the hardware performance fingerprint library, and realize the fast start of the scheduling service.

[0043] In some embodiments, the batch encoder obtains a long-short classification threshold, compares the sequence length of the token sequence of each inference task with the long-short classification threshold, and schedules the inference task with the sequence length greater than the long-short classification threshold to the long task queue, and schedules the inference task with the sequence length less than or equal to the long-short classification threshold to the short task queue.

[0044] In some embodiments, the task buffer is used to buffer the inference tasks, and contains the long task queue and the short task queue, and follows the first-in first-out principle. The batch encoder inserts the inference tasks with different lengths into the long task queue and the short task queue.

[0045] In step 1014, the task state of the inference task is updated to be processed.

[0046] In some embodiments, the batch encoder inserts the inference tasks with different lengths into the long task queue and the short task queue and marks the task state as to be processed.

[0047] The traditional inference task type has poor adaptability, different inference tasks (long input / short input, long output / short output, etc.) are friendly to different processors, and the traditional fixed ratio scheduling strategy cannot allocate tasks according to the type of inference task.

[0048] The embodiment of the present application considers that the second processor inference service has short input affinity, and is more suitable for processing short input inference tasks. According to the service level objective (SLO) index set by the user, the long-short task threshold of the inference task is calculated, and the inference tasks with different lengths are scheduled to the long task queue and the short task queue according to the long-short task threshold. According to the different long-short types of the inference tasks, classification is performed to realize the adaptability of the inference task type. The embodiment splits and encodes the single inference request and multiple inference tasks, classifies the inference tasks with similar input lengths, increases the time of GPU / CPU concurrent inference calculation, realizes reasonable long / short inference task scheduling, and can fully exert the calculation performance of GPU / CPU.

[0049] At step 102, when it is monitored that there are inference tasks to be processed in the two task queues, the working state of the first processor inference service is judged based on the first task number and the first processor maximum concurrency, the first processor maximum concurrency is obtained based on the current conditions of the two task queues, and the first task number is the number of tasks in the two task queues that are being processed by the first processor inference service.

[0050] In this embodiment, the scheduling principle of the inference task scheduling includes: the first processor resource is idle, and the inference task is preferentially scheduled to the first processor inference service, and the second processor inference service has short input affinity, and is more suitable for processing inference tasks in the short task queue. Therefore, in this embodiment, the working state of the first processor inference service is first obtained, and the working state includes an idle state and a non-idle state.

[0051] In some embodiments, the heterogeneous scheduling service monitors in real time whether there are inference tasks in the task state of the long task queue and the short task queue to be processed, and if there are inference tasks to be processed, the scheduling processing is performed.

[0052] In some embodiments, the current conditions of the two task queues include the first average length of the input tokens of the current inference tasks in the two task queues.

[0053] In some embodiments, the method of judging the working state of the first processor inference service based on the first task number and the first processor maximum concurrency includes: Figure 5 As shown in the figure, the method includes the following steps: At step 1021, the first average length of the input tokens of the current inference tasks in the two task queues and the first inference performance data corresponding to the first processor inference service are obtained.

[0054] In this embodiment, the inference tasks in the long task queue and the short task queue have multiple task states, including to be processed, being processed by the first processor, being processed by the second processor, and being completed. The number of inference tasks in the two task queues whose task state is marked as being processed by the first processor is obtained, that is, the first task number, also referred to as the first processor processing task number. .

[0055] In this embodiment, the first processor inference service can process inference tasks in the long task queue and the short task queue, and therefore, the first average length of the input tokens of the current inference tasks in the two task queues is obtained.

[0056] At step 1022, the first processor maximum concurrency is obtained based on the user service level target index, the first inference performance data, and the first average length.

[0057] In some embodiments, the first processor maximum concurrency is obtained based on the user service level target indicator, the first inference performance data and the first average length by the following formula :

[0058] wherein: represents the maximum delay of the first word piece, represents the maximum interval between the generation of each word piece, represents the maximum pre-filling throughput under the condition that the inference performance data corresponding to the first processor inference service satisfies ; represents the first average length of the input word pieces of the inference tasks in the two task queues; represents the maximum concurrency under the condition that the first inference performance data corresponding to the first processor inference service satisfies ; and is a safety factor, and the default value is 0.95, in order to prevent overload.

[0059] The embodiment designs a dynamic first processor maximum concurrency calculation formula, dynamically calculates the first processor maximum concurrency under the current inference task condition according to the inference tasks in the current long / short task queue, solves the problem that the fixed concurrency cannot cope with multiple concurrent inference tasks, and can maximize the first processor concurrency to the fullest extent, and maximizes the inference performance.

[0060] In step 1023, it is judged whether the first processor maximum concurrency is greater than the first task number. If it is greater, it is judged that the working state of the first processor inference service is an idle state.

[0061] In the embodiment, if the first processor maximum concurrency is greater than the first task number , it is considered that the first processor inference service is in an idle state, otherwise, it is considered that the first processor inference service is in a non-idle state.

[0062] The embodiment does not need to monitor the utilization rate of the processor and the inference service concurrency, and can obtain the first processor maximum concurrency at the current time from the task queue while scheduling the inference task, avoiding the error scheduling problem caused by the asynchronization of performance monitoring and scheduling decision time. In addition, the first processor maximum concurrency under the current inference task condition is dynamically calculated according to the tasks in the current long / short task queue, solving the problem that the fixed concurrency cannot cope with multiple concurrent inference tasks, and maximizing the processor concurrency to the fullest extent, and maximizing the inference performance.

[0063] In step 103, based on the working state of the first processor inference service, the to-be-processed inference task is scheduled to the first processor inference service or the second processor inference service.

[0064] In some embodiments, the implementation method of scheduling the to-be-processed inference task to the first processor inference service or the second processor inference service based on the working state of the first processor inference service includes the following steps: Figure 6 In step 1031, if the working state of the first processor inference service is the idle state, a first difference value M between the first processor maximum concurrency and the first task number is obtained.

[0065] In this embodiment, the first difference value M is In order to obtain M to-be-processed inference tasks from the long task queue and the short task queue and schedule them to the first processor inference service.

[0066] In step 1032, if the to-be-processed inference tasks exist in the two task queues respectively, and the number of to-be-processed inference tasks in the long task queue is greater than or equal to M, M to-be-processed inference tasks in the long task queue are obtained and scheduled to the first processor inference service, and the task states of the M inference tasks are updated to processing.

[0067] In this embodiment, another scheduling principle for the first processor inference service is that the scheduling priority is long task queue task>short task queue task, that is, the first processor inference service preferentially processes long inference task types.

[0068] In step 1033, if the number of to-be-processed inference tasks in the long task queue is less than M, M to-be-processed inference tasks are obtained from the two task queues and scheduled to the first processor inference service, the M to-be-processed inference tasks include all to-be-processed inference tasks in the long task queue, and the task states of the M inference tasks are updated to processing.

[0069] If the number of to-be-processed inference tasks in the long task queue is less than , all to-be-processed inference tasks in the long task queue and part of inference tasks in the short task queue are obtained, M to-be-processed inference tasks are obtained, and are scheduled to the first processor inference service.

[0070] In step 1034, if the working state of the first processor inference service is the non-idle state, it is judged whether the to-be-processed inference task is located in the short task queue.

[0071] ​​Because the second processor inference service has short input affinity, it is more suitable for processing inference tasks in a short task queue, and in the case that the first processor inference service is in an idle state, it is also necessary to determine whether the inference task to be processed is a short inference task.

[0072] If yes, a second number of tasks, a second average length of input tokens of a current inference task in the short task queue and second inference performance data are obtained, wherein the second number of tasks is a number of tasks in the short task queue that are being processed by the second processor inference service.

[0073] If the inference task to be processed is a short inference task, it is necessary to determine whether the current state of the second processor can process the inference task. To determine whether the current state of the second processor can process the inference task, the current second processor maximum concurrency needs to be obtained, and thus the second average length of input tokens of the current inference task in the short task queue and the second inference performance data need to be obtained.

[0074] In step 1036, the second processor maximum concurrency is obtained based on the user service level objective indicator, the second inference performance data and the second average length.

[0075] In some embodiments, the second processor maximum concurrency is obtained by the following formula :

[0076] wherein: represents the maximum delay of the first token, represents the maximum interval between each token, represents the maximum prefill throughput in the inference performance data corresponding to the second processor inference service under the condition that represents the second average length of input tokens of the inference task in the short task queue; represents the maximum concurrency in the inference performance data corresponding to the second processor inference service under the condition that represents the second average length of input tokens of the inference task in the short task queue; represents the maximum concurrency in the inference performance data corresponding to the second processor inference service under the condition that and are safety factors.

[0077] The embodiment designs a calculation formula of dynamic second processor maximum concurrency, dynamically calculates the second processor maximum concurrency under the current inference task condition according to the inference task of the current short task queue, solves the problem that the fixed concurrency cannot cope with various concurrent inference tasks, maximizes the second processor concurrency, maximizes the inference performance, and introduces scheduling decisions under the SLO indicator, realizes that different processor heterogeneous computing power is fully utilized on the basis of guaranteeing that the inference performance experience of a user is not affected, and the inference performance in a resource restricted environment is improved.

[0078] At step 1037, if the second number of tasks is less than the maximum concurrency of the second processor, a second difference N between the maximum concurrency of the second processor and the second number of tasks is obtained, N inference tasks to be processed in the short task queue are scheduled to the second processor inference service, and the task states of the N inference tasks to be processed are updated to processing.

[0079] In this embodiment, according to the different affinity of different inference task types to the first processor and the second processor, when the load of the first processor increases, more short inference tasks friendly to the second processor are scheduled to the second processor, so as to reduce the difference between the inference performance of the first processor and the second processor.

[0080] At step 1038, inference tasks with a task state of completed are obtained from the two task queues, and the inference tasks with the task state of completed are deleted from the two task queues.

[0081] In some embodiments, the decoder obtains inference tasks with a task state of completed from the long task queue and the short task queue, and deletes the inference tasks with the task state of completed from the long task queue and the short task queue.

[0082] At step 1039, the inference result corresponding to the completed inference task is decoded to obtain a decoded inference result, and the decoded inference results belonging to the same unique request identifier are assembled into a batch processing result, and the batch processing result is returned to the user end.

[0083] In some embodiments, the decoder decodes the inference result of the completed inference task using the tokenizer of the inference model to convert the wordpiece data into text data, and then assembles the inference results of the same request ID into a batch processing result and returns the batch processing result to the user.

[0084] The heterogeneous computing power scheduling method of the embodiment of the present application realizes the affinity mechanism of long and short inference task types and different processors by splitting the inference request into different long and short inference task types, preferentially schedules the inference tasks of different processor affinities to the inference service of the corresponding processor, and reduces the difference between the inference performance of different processors. By introducing long / short task queues, different types of inference tasks are more reasonably scheduled to the inference service of different processors, maximizing the use of the mixed computing power of heterogeneous processors and improving the overall throughput of the inference service. At the same time, it is realized that there is no need to monitor the utilization rate and the concurrent amount of the inference service of the processor, and the concurrent amount of the processor inference service at the current time can be obtained from the task queue at the same time as the inference task scheduling, avoiding the problem of incorrect scheduling caused by the asynchronization of performance monitoring and scheduling decision time. According to the tasks in the current long / short task queue, the maximum concurrency of the processor under the current inference task condition is dynamically calculated, which solves the problem that the fixed concurrency cannot cope with various concurrent inference tasks, and can maximize the concurrency of the processor, thereby maximizing the inference performance. The dynamic scheduling process is realized, and the fixed proportion allocation method cannot adapt to the dynamic load change of the inference scene. The mixed computing power resources of the heterogeneous processor are fully utilized, and the problem of idle and waste of the second processor resources while the first processor of the AI inference server runs at high load is solved.

[0085] In order to clearly illustrate the above-mentioned embodiments, specific examples will be described. Figure 3 The scheduling flowchart of the heterogeneous scheduling service of the heterogeneous computing power scheduling system provided by the embodiment of the present application is shown in FIG. 1. Figure 3 As shown in the figure, the heterogeneous inference computing power scheduling method of the present application is applied to a heterogeneous computing power scheduling system, which is configured with a batch processing encoder, a task buffer, a scheduler, a hardware performance fingerprint library and a batch processing decoder.

[0086] When the heterogeneous scheduling service starts, the scheduler first detects the GPU and CPU models of the current environment, and then obtains the inference performance data of the corresponding models from the built-in hardware performance fingerprint library. When the inference performance data of the corresponding GPU and CPU models cannot be detected, the GPU / CPU performance test program is started to test the inference performance of the GPU and CPU models of the current environment. After the test is completed, the obtained inference performance data is stored in the hardware fingerprint library. The heterogeneous scheduling service receives the inference request sent by the user, and the inference request first enters the batch processing encoder for preprocessing.

[0087] The processing flow of the batch processing encoder is shown in FIG. 3. Figure 7As shown, the prompt of the inference request is split, the tokenizer of the inference model is used to encode the prompt, the text is converted into tokens, the length of the tokens is calculated, the long and short classification threshold calculated by the scheduler is used for distribution, and the long and short task queues in the task buffer are saved respectively, and each inference task is bound to the ID of the inference request.

[0088] The scheduler obtains the inference tasks to be processed from the long and short task queues and marks the task state as GPU processing. After the inference is completed, the inference result is saved to the corresponding task data table and the task state is updated to completed. The batch decoding decoder obtains the tasks with the completed state from the long and short task queues and deletes the completed tasks from the long and short task queues.

[0089] The scheduler is the control center of the heterogeneous scheduling service. When the scheduling service is started, the scheduler obtains the hardware device GPU and CPU information of the current environment, then finds the inference performance data of the corresponding hardware device from the hardware performance fingerprint library. If there is no hardware performance fingerprint library, the performance test program is triggered to test the CPU and GPU inference performance, and the test result is saved to the hardware performance fingerprint library. Then, the scheduling strategy is calculated according to the service level (SLO) index configured by the user, and then the inference task is obtained from the task buffer according to the scheduling strategy and is scheduled to the GPU / CPU inference service. The performance test process of the CPU and GPU inference performance is as shown in Figure 8 As shown, first, the IP and port of the CPU inference service and the IP and port of the GPU inference service are configured, then the streaming request configuration is performed, the OpenAI API interface of the CPU inference service and the GPU inference service is called respectively, the streaming response is listened to, the first token arrival time TTFT and the subsequent token arrival time are recorded in the test code, the average time interval TPOT is calculated, and after the test is completed, the performance data is calculated according to the input data, the return data and the recorded time points.

[0090] The scheduling logic of the scheduler is as shown in Figure 9 The scheduler monitors whether there is an inference task to be processed in the two task queues in real time, and if there is, the scheduling calculation is performed. First, the number of GPU processing tasks in the two task queues is queried , then the maximum GPU concurrency under the current task queue condition is calculated , if is less than , it means that the GPU resource is idle at this time, and the inference task is preferentially scheduled to the GPU inference service, and the number of scheduled inference tasks is , the scheduling priority is that the inference task in the long task queue is greater than the inference task in the short task queue, and the scheduling order is FIFO (First In First Out). If the number is greater than or equal to , it indicates that there is no free GPU resource for testing, and the inference task is scheduled to the CPU inference service. First, it is necessary to determine whether the inference task to be processed is in the short task queue. If so, the task type of the CPU inference service is satisfied, scheduling is performed, the number of tasks in the CPU processing in the short task queue is queried , and the maximum CPU concurrency under the current short task queue condition is calculated . If is less than , the number of scheduled tasks is tasks to the CPU inference service, and the scheduling order is FIFO.

[0091] The batch decoding decoder is used to process the inference task completed by inference. The processing flow is as shown in Figure 10 , the inference task with the completed state is obtained from the two task queues of the task buffer, the completed inference task is deleted from the task queue while the completed inference task is obtained, and then the tokenizer of the inference model is used to decode the inference result of the completed inference task. The token data is converted into text data, and then the inference results of the same request ID are assembled into a batch processing result and returned to the user.

[0092] By reasonably designing the heterogeneous computing power scheduling system composed of the batch processing encoder, the batch processing decoder, the task buffer, and the scheduler and other functional modules, the inference tasks are reasonably allocated, the GPU+CPU hybrid computing power is maximized, and the overall throughput of the inference service is improved.

[0093] In order to realize the heterogeneous inference computing power scheduling method provided by the embodiments of the present disclosure, the embodiments of the present disclosure further provide a heterogeneous inference computing power scheduling device. The device is configured in a heterogeneous computing power scheduling system, the system includes a first processor inference service and a second processor inference service, and the system is configured with two task queues when starting. The two task queues include a long task queue and a short task queue. As shown in Figure 11 , it includes: A task division module 1111 is configured to receive an inference request, split the inference request to obtain at least one inference task, and schedule the inference task to the long task queue or the short task queue according to the encoding length of each inference task. The state obtaining module 1112 is configured to, when it is monitored that there are inference tasks to be processed in the two task queues, determine a working state of the first processor inference service based on a first task quantity and a first processor maximum concurrency, the first processor maximum concurrency being obtained based on current conditions of the two task queues, and the first task quantity being a quantity of tasks in the two task queues that are being processed by the first processor inference service. The task scheduling module 1113 is configured to, based on the working state of the first processor inference service, schedule the inference tasks to be processed to the first processor inference service or the second processor inference service.

[0094] Further, in a possible implementation manner of the embodiment of the present disclosure, the task division module 1111 is specifically configured to: divide the inference request into at least one inference task according to prompt words included in the inference request, and bind the at least one inference task to a unique request identifier of the inference request, a quantity of the at least one inference task being the same as a quantity of the prompt words included in the inference request; encode each inference task in the at least one inference task into a wordpiece sequence, and obtain a sequence length of the wordpiece sequence; schedule an inference task with a sequence length greater than a long-short classification threshold to the long task queue, and schedule an inference task with a sequence length less than or equal to the long-short classification threshold to the short task queue, wherein the long-short classification threshold is obtained based on a user service level target indicator and second inference performance data of the second processor inference service in a system initialization phase; update a task state of the inference task to be processed.

[0095] Further, in a possible implementation manner of the embodiment of the present disclosure, the current conditions of the two task queues include a first average length of input wordpieces of current inference tasks in the two task queues, and the state obtaining module 1112 is specifically configured to: obtain the first average length of the input wordpieces of the current inference tasks in the two task queues and first inference performance data corresponding to the first processor inference service; obtain the first processor maximum concurrency based on the user service level target indicator, the first inference performance data, and the first average length; determine whether the first processor maximum concurrency is greater than the first task quantity, and if so, determine that the working state of the first processor inference service is an idle state.

[0096] Further, in a possible implementation manner of the embodiment of the present disclosure, the user service level target indicator includes a maximum delay of a first wordpiece and a maximum interval between generation of each wordpiece, and when obtaining the first processor maximum concurrency based on the user service level target indicator, the first inference performance data, and the first average length, the state obtaining module 1112 is specifically configured to: By the first formula, based on the user service level target indicator, the first inference performance data and the first average length, the first processor maximum concurrency , the first formula is expressed as follows:

[0097] Wherein: represents the first word maximum delay, represents the maximum interval between generating each word, represents the maximum prefill throughput under the condition that satisfies in the first processor inference service corresponding to the first inference performance data; , represents the first average length of the input word of the inference task in the two task queues; represents the maximum concurrency under the condition that satisfies in the first processor inference service corresponding to the first inference performance data; and is a safety factor.

[0098] Further, in a possible implementation manner of the embodiment of the present disclosure, the user service level target indicator includes the first word maximum delay, and when the task division module 1111 acquires the long-short classification threshold, specifically used for: Acquire the maximum input length satisfying the first word maximum delay in the second inference performance data, and take the maximum input length as the long-short classification threshold.

[0099] Further, in a possible implementation manner of the embodiment of the present disclosure, the apparatus further includes a performance test module 1114, configured to: Acquire the first processor information corresponding to the first processor inference service and the second processor information corresponding to the second processor inference service; Based on the first processor information and the second processor information, determine whether the first inference performance data and the second inference performance data are stored in the first storage structure; If not, configure the IP and port of the second processor inference service and the IP and port of the first processor inference service; Call the OpenAI API interface of the second processor inference service and the first processor inference service to perform inference performance test, and acquire the first inference performance data and the second inference performance data; Store the first inference performance data and the second inference performance data into the first storage structure.

[0100] Further, in a possible implementation manner of the embodiment of the present disclosure, the task scheduling module 1113 is specifically configured to: If the working state of the first processor inference service is the idle state, a first difference value M between the first processor maximum concurrency and the first task number is obtained; If the to-be-processed inference tasks exist in the two task queues respectively, and the number of the to-be-processed inference tasks in the long task queue is greater than or equal to M, M to-be-processed inference tasks in the long task queue are obtained and scheduled to the first processor inference service, and the task states of the M inference tasks are updated to processing. If the number of the to-be-processed inference tasks in the long task queue is less than M, M to-be-processed inference tasks are obtained and scheduled to the first processor inference service from the two task queues, the M to-be-processed inference tasks include all the to-be-processed inference tasks in the long task queue, and the task states of the M inference tasks are updated to processing.

[0101] Further, in a possible implementation of the embodiment of the present disclosure, the task scheduling module 1113 is further configured to: If the working state of the first processor inference service is the non-idle state, it is determined whether the to-be-processed inference task is located in the short task queue; If yes, a second task number is obtained, the second task number being the number of the tasks in processing in the short task queue by the second processor inference service; The second average length of the input tokens of the current inference task in the short task queue and the second inference performance data are obtained; The second processor maximum concurrency is obtained based on the user service level target index, the second inference performance data and the second average length; If the second task number is less than the second processor maximum concurrency, a second difference value N between the second processor maximum concurrency and the second task number is obtained; N to-be-processed inference tasks in the short task queue are scheduled to the second processor inference service, and the task states of the N to-be-processed inference tasks are updated to processing.

[0102] Further, in a possible implementation of the embodiment of the present disclosure, the task scheduling module 1113 is further configured to: The inference tasks with the task states of completed are obtained from the two task queues, and the inference tasks with the task states of completed are deleted from the two task queues; The inference results corresponding to the inference tasks with the task states of completed are decoded to obtain decoded inference results, the decoded inference results belonging to the same unique request identifier are assembled into a batch processing result, and the batch processing result is returned to the user end.

[0103] It should be noted that the foregoing explanation and description of the embodiment of the heterogeneous inference computing power scheduling method are also applicable to the embodiment of the heterogeneous inference computing power scheduling device, which will not be described here again.

[0104] Embodiments of the present disclosure also provide an electronic device, comprising a memory and a processor, the memory storing a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-mentioned heterogeneous reasoning computing power scheduling method embodiments.

[0105] Embodiments of the present disclosure also provide a computer-readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above-mentioned heterogeneous reasoning computing power scheduling method embodiments when running.

[0106] In an example embodiment, the above-mentioned computer-readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0107] Embodiments of the present disclosure also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned heterogeneous reasoning computing power scheduling method embodiments.

[0108] Embodiments of the present disclosure also provide another computer program product, which comprises a non-volatile computer-readable storage medium, and the non-volatile computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned heterogeneous reasoning computing power scheduling method embodiments.

[0109] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0110] The above has introduced in detail a target detection method provided by the present disclosure. The principles and implementation manners of the present disclosure have been described by applying specific examples, and the above description of the examples is only used to help understand the method of the present disclosure and its core idea. It should be pointed out that, for those skilled in the art, without departing from the principles of the present disclosure, the present disclosure can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the claims of the present disclosure.

Claims

1. A heterogeneous inference compute power scheduling method, characterized in that, The method is applied to a heterogeneous computing power scheduling system, the system comprising a first processor inference service and a second processor inference service, the system being configured with two task queues at startup, the two task queues comprising a long task queue and a short task queue, the method comprising: receiving an inference request, splitting the inference request to obtain at least one inference task, and scheduling the inference tasks to the long task queue or the short task queue according to the encoding length of each inference task; when it is monitored that there is an inference task to be processed in the two task queues, judging the working state of the first processor inference service based on a first task number and a first processor maximum concurrency, the first processor maximum concurrency being obtained based on the current conditions of the two task queues, and the first task number being the number of tasks in the two task queues being processed by the first processor inference service; based on the working state of the first processor inference service, scheduling the inference task to be processed to the first processor inference service or the second processor inference service.

2. The method of claim 1, wherein, The splitting of the inference request to obtain at least one inference task; and scheduling the inference tasks to the long task queue or the short task queue according to the encoding length of each inference task, comprises: dividing the inference request into at least one inference task according to the prompt word division identifier included in the inference request, and binding the at least one inference task with the unique request identifier of the inference request, the number of the at least one inference task being the same as the number of the prompt words included in the inference request; encoding each of the at least one inference task into a word sequence, and obtaining the sequence length of the word sequence; scheduling the inference task with the sequence length greater than a long-short classification threshold to the long task queue, and scheduling the inference task with the sequence length less than or equal to the long-short classification threshold to the short task queue, wherein the long-short classification threshold is obtained based on a user service level target indicator and second inference performance data of the second processor inference service in a system initialization stage; updating the task state of the inference task to be processed.

3. The method according to claim 1 or 2, characterized in that, The current conditions of the two task queues include a first average length of input word units of current inference tasks in the two task queues, The judging of the working state of the first processor inference service based on a first task number and a first processor maximum concurrency, comprises: obtaining a first average length of input word units of current inference tasks in the two task queues and first inference performance data corresponding to the first processor inference service; obtaining the first processor maximum concurrency based on a user service level target indicator, the first inference performance data and the first average length; judging whether the first processor maximum concurrency is greater than the first task number, and if so, judging the working state of the first processor inference service as an idle state.

4. The method of claim 3, wherein, The user service level target indicator comprises a maximum delay of a first word unit and a maximum interval between generating each word unit. obtaining the first processor maximum concurrency based on the user service level target indicator, the first inference performance data and the first average length, comprising: By the first formula, based on the user service level target indicator, the first inference performance data, and the first average length, the first processor maximum concurrency is derived, the first formula being expressed as follows: wherein: represents the maximum latency of the first word, represents the maximum interval between each word generated, represents the maximum pre-populated throughput under the condition that is satisfied in the first processor inference service corresponding first inference performance data; represents the first average length of the inference task input word in the long task queue and the short task queue; represents the maximum concurrency under the condition that is satisfied in the inference performance data corresponding to the first processor inference service; and is a safety factor.

5. The method of claim 2, wherein, the user service level target indicator comprises a first word element maximum delay, and the method for obtaining the long-short classification threshold value, comprising: obtaining the maximum input length in the second inference performance data that meets the first word element maximum delay; the maximum input length is taken as the long-short classification threshold value.

6. The method of claim 1, wherein, the method further comprises: obtaining the first processor information corresponding to the first processor inference service and the second processor information corresponding to the second processor inference service; based on the first processor information and the second processor information, it is judged whether the first inference performance data and the second inference performance data are stored in the first storage structure; if not, the IP and port of the second processor inference service and the IP and port of the first processor inference service are configured; the OpenAI API interface of the second processor inference service and the first processor inference service is called to perform inference performance test, and the first inference performance data and the second inference performance data are obtained; the first inference performance data and the second inference performance data are stored in the first storage structure.

7. The method of claim 3, wherein, based on the working state of the first processor inference service, the inference task to be processed is scheduled to the first processor inference service or the second processor inference service, comprising: if the working state of the first processor inference service is an idle state, a first difference M between the first processor maximum concurrency and the first task number is obtained; if the inference tasks to be processed exist in the two task queues respectively, and the number of inference tasks to be processed in the long task queue is greater than or equal to M, M inference tasks to be processed are obtained from the long task queue and scheduled to the first processor inference service, and the task state of the M inference tasks is updated to processing. if the number of inference tasks to be processed in the long task queue is less than M, M inference tasks to be processed are obtained from the two task queues and scheduled to the first processor inference service, the M inference tasks to be processed include all inference tasks to be processed in the long task queue, and the task state of the M inference tasks is updated to processing.

8. The method of claim 1, wherein, based on the working state of the first processor inference service, the inference task to be processed is scheduled to the first processor inference service or the second processor inference service, comprising: if the working state of the first processor inference service is a non-idle state, it is judged whether the inference task to be processed is located in the short task queue; if yes, the second task number, the second average length of the input word element of the current inference task in the short task queue and the second inference performance data are obtained, wherein the second task number is the number of tasks in the short task queue processed by the second processor inference service; obtaining the second processor maximum concurrency based on the user service level target indicator, the second inference performance data and the second average length. If the second task number is less than the second processor maximum concurrency, a second difference N between the second processor maximum concurrency and the second task number is obtained; The N inference tasks to be processed in the short task queue are scheduled to the second processor inference service, and task states of the N inference tasks to be processed are updated to processing.

9. The method of claim 2, wherein, The method further comprises: An inference task with a completed task state is obtained from the two task queues, and the inference task with the completed task state is deleted from the two task queues; A decoding process is performed on an inference result corresponding to the inference task with the completed task state, a decoded inference result is obtained, the decoded inference results belonging to the same unique request identifier are assembled into a batch processing result, and the batch processing result is returned to a user end.

10. An electronic device, comprising: Comprise: A memory for storing a computer program; A processor for executing the computer program to implement the steps in any one of claims 1 to 9.

Images