Heterogeneous inference computing power scheduling method and device, and storage medium

Abstract

The present disclosure provides a heterogeneous reasoning computing power scheduling method, device and storage medium. A reasoning request is received, the reasoning request is split to obtain at least one reasoning task, and the reasoning tasks are scheduled to a long task queue or a short task queue according to the encoding length of each reasoning task. When it is monitored that there is a reasoning task in a state of to-be-processed in the long task queue and the short task queue, the working state of a first processor reasoning service is judged based on the number of tasks in the first processor processing in the long task queue and the short task queue and the maximum concurrency of the first processor, and the maximum concurrency of the first processor is obtained based on the current conditions of the long task queue and the short task queue. Based on the working state of the first processor reasoning service, the to-be-processed reasoning task is scheduled to the first processor reasoning service or a second processor reasoning service, so as to maximize the reasonable use of the mixed computing power resources of the heterogeneous processor.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of large model inference deployment, in particular to a heterogeneous inference computing power scheduling method and device and storage medium. BACKGROUND

[0002] In the scheduling problem of AI inference large model inference computing power, the graphics processing unit (GPU) has become the mainstream inference platform due to its parallel computing advantage. The larger the number of inference model parameters and the more the number of concurrent inference requests, the more GPU computing power resources are needed. However, GPU resources are expensive, and constantly increasing GPU resources may result in huge cost investment. The central processing unit (CPU) deployed on the AI inference server is often in an idle state, resulting in waste of computing resources of the AI inference server. CPU and GPU hybrid heterogeneous computing power inference can be used to solve the problem of idle CPU resources. However, the current CPU and GPU hybrid heterogeneous computing power inference cannot effectively maximize the rational use of GPU and CPU hybrid computing power resources. SUMMARY

[0003] The present disclosure provides a heterogeneous inference computing power scheduling method, device and system, which can maximize the rational use of hybrid computing power resources of heterogeneous processors.

[0004] To solve the above technical problems, the present disclosure provides a heterogeneous inference computing power scheduling method, which is applied to a heterogeneous computing power scheduling system, the system including a first processor inference service and a second processor inference service, the system being configured with two task queues at startup, the two task queues including a long task queue and a short task queue, the method including:

[0005] receiving an inference request, splitting the inference request to obtain at least one inference task, and scheduling the inference tasks to the long task queue or the short task queue according to the encoding length of each inference task;

[0006] when it is monitored that there are inference tasks to be processed in the two task queues, judging the working state of the first processor inference service based on a first task number and a first processor maximum concurrency, the first processor maximum concurrency being obtained based on the current conditions of the two task queues, and the first task number being the number of tasks in the two task queues being processed by the first processor inference service;

[0007] scheduling the inference tasks to be processed to the first processor inference service or the second processor inference service based on the working state of the first processor inference service.

[0008] This disclosure also provides a heterogeneous inference computing power scheduling device, the device being configured in a heterogeneous computing power scheduling system, the system including a first processor inference service and a second processor inference service, the system configuring two task queues upon startup, the two task queues including a long task queue and a short task queue, the device comprising:

[0009] The task partitioning module is used to receive inference requests, split the inference requests to obtain at least one inference task, and schedule the inference tasks to the long task queue or the short task queue according to the encoding length of each inference task.

[0010] The status acquisition module is used to determine the working status of the first processor inference service based on the first number of tasks and the maximum concurrency of the first processor when it detects that there are pending inference tasks in the two task queues. The first maximum concurrency of the first processor is obtained based on the current conditions of the two task queues. The first number of tasks is the number of tasks being processed by the first processor inference service in the two task queues.

[0011] The task scheduling module is used to schedule the pending inference task to the first processor inference service or the second processor inference service based on the working status of the first processor inference service.

[0012] This disclosure also provides an electronic device, including:

[0013] Memory, used to store computer programs;

[0014] A processor is configured to implement the steps of any of the heterogeneous inference computing power scheduling methods provided in the embodiments of this disclosure when executing a computer program.

[0015] This disclosure also provides a computer-readable storage medium storing a computer program, wherein when executed by a processor, the computer program implements the steps of any of the heterogeneous inference computing power scheduling methods provided in this disclosure.

[0016] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the heterogeneous inference computing power scheduling methods provided in this disclosure.

[0017] This disclosure provides a heterogeneous inference computing power scheduling method. By splitting inference requests into inference task types of different lengths, it implements an affinity mechanism between inference task types and different processors, prioritizing inference tasks with different processor affinities to the inference services of the corresponding processors, thus reducing the performance differences between different processors. By introducing long / short task queues, it more rationally schedules different types of inference tasks to inference services on different processors, maximizing the utilization of the hybrid computing power of heterogeneous processors and improving the overall throughput of the inference service. Simultaneously, it eliminates the need to monitor processor utilization and inference service concurrency; the concurrency status of the processor's inference service at the current moment can be synchronously obtained from the task queue during inference task scheduling, avoiding erroneous scheduling problems caused by the asynchrony between performance monitoring and scheduling decision time. Based on the tasks in the current long / short task queues, it dynamically calculates the maximum concurrency of the processor under the current inference task conditions, solving the problem that a fixed concurrency level cannot handle multiple concurrent inference tasks, maximizing the processor's concurrency and improving inference performance. It implements a dynamic scheduling process, solving the problem that fixed-ratio allocation methods cannot adapt to inference scenarios with dynamically changing loads. By making full use of the hybrid computing resources of heterogeneous processors, the problem of the first processor of the AI ​​inference server running under high load while the second processor's resources are idle and wasted is solved.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a flowchart illustrating a heterogeneous inference computing power scheduling method provided in an embodiment of this disclosure;

[0021] Figure 2 This is a flowchart illustrating another heterogeneous inference computing power scheduling method provided in an embodiment of this disclosure;

[0022] Figure 3 This is a schematic diagram of the scheduling process for the heterogeneous scheduling service provided in the embodiments of this disclosure;

[0023] Figure 4 This is a flowchart illustrating another heterogeneous inference computing power scheduling method provided in an embodiment of this disclosure;

[0024] Figure 5 A flowchart illustrating yet another heterogeneous inference computing power scheduling method provided in this embodiment of the present disclosure;

[0025] Figure 6 A flowchart of yet another heterogeneous inference computing power scheduling method provided in an embodiment of this disclosure;

[0026] Figure 7 This is a flowchart of the batch encoder processing provided in the embodiments of this disclosure;

[0027] Figure 8 A flowchart illustrating the inference performance test of the scheduler provided in the embodiments of this disclosure;

[0028] Figure 9 A flowchart illustrating the inference task scheduling process performed by the scheduler provided in this embodiment of the disclosure;

[0029] Figure 10 This is a flowchart of the batch decoder processing provided in the embodiments of this disclosure;

[0030] Figure 11 This is a schematic diagram of another heterogeneous inference computing power scheduling device provided in an embodiment of the present disclosure. Detailed Implementation

[0031] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0032] In related technologies, current CPU+GPU hybrid heterogeneous computing power inference is still under development. The mainstream scheduling strategies currently involve dividing the workload into fixed proportions or scheduling based on GPU / CPU resource utilization. For example, 80% of inference requests are scheduled to the GPU and 20% to the CPU, or requests are scheduled to the GPU when it is idle and to the CPU when it is fully loaded. However, real-world inference scenarios are often complex, with dynamic changes in request concurrency, inference hint length, and CPU / GPU computing status. Static proportions cannot effectively schedule inference tasks dynamically and cannot adapt to inference scenarios with dynamically changing loads. Scheduling based solely on GPU / CPU resource utilization cannot achieve true... Real-time monitoring and concurrency control can easily lead to the detection of resource utilization rates that are not the actual resource utilization rates at the current scheduling moment. This can result in incorrect scheduling of inference tasks, failing to maximize the utilization of GPU+CPU computing power and potentially reducing inference performance. This is because hardware utilization detection and scheduling decisions are not synchronized. Hardware utilization detection is periodic, and the hardware utilization data obtained during scheduling decisions is often not the actual data at the current moment. This delay can lead to incorrect scheduling judgments. For example, even if the GPU inference service has idle resources, the inference task may be incorrectly scheduled to the CPU inference service because the previous hardware detection showed that the resource was not idle. This results in the overall inference performance being lower than that of a pure GPU inference service. Furthermore, the concurrent data collection and scheduling decision-making times are not synchronized. The collection of concurrency data for the inference service has periodic and delayed characteristics. The concurrency number of the inference service obtained during scheduling decisions is often not the actual concurrency number at the current moment. This delay can lead to incorrect scheduling judgments, resulting in a mismatch between the concurrent tasks scheduled to the GPU / CPU and the current idle concurrency number of the GPU / CPU. This may cause a large number of requests to be blocked or the concurrency number to fall below the maximum concurrency number, thus failing to effectively maximize the utilization of the combined computing power of GPU and CPU.

[0033] Therefore, current hybrid heterogeneous computing power inference using CPUs and GPUs cannot maximize the overall computing resources of the server while meeting the Service Level Objective (SLO) metric.

[0034] In view of this, this application provides a CPU-GPU hybrid heterogeneous computing power inference method, system and storage medium under computing resource-constrained environment, which can maximize the rational use of GPU+CPU hybrid computing power and improve the overall throughput of inference service.

[0035] The heterogeneous inference computing power scheduling method, apparatus, and storage medium of this disclosure are described below with reference to the accompanying drawings.

[0036] Figure 1This is a flowchart illustrating a heterogeneous inference computing power scheduling method according to an embodiment of this disclosure, as shown below. Figure 1 As shown, the method may include the following steps:

[0037] Step 101: Receive inference requests, split the inference requests to obtain at least one inference task, and schedule the inference tasks to the long task queue or the short task queue according to the encoding length of each inference task.

[0038] The heterogeneous inference computing power scheduling method of this application embodiment is applied to a heterogeneous computing power scheduling system, which includes a first processor inference service (such as a GPU inference service) and a second processor inference service (such as a CPU inference service). The system is configured with two task queues at startup, including a long task queue and a short task queue.

[0039] In one example, such as Figure 2 The diagram illustrates a hybrid heterogeneous computing power scheduling system under a computing resource-constrained environment. This system includes a heterogeneous scheduling service, a GPU inference service, and a CPU inference service. The heterogeneous scheduling service, located as an intermediate layer above the GPU and CPU inference services, receives inference requests from users and selects whether to schedule the request to the GPU or CPU inference service based on the calculation results of the scheduling decision algorithm. The GPU inference service is used to invoke the large model inference service powered by GPU computing power, receives inference requests scheduled by the heterogeneous scheduling service, and uses GPU computing power to calculate inference results. The CPU inference service is used to invoke the large model inference service powered by CPU computing power, receives inference requests scheduled by the heterogeneous scheduling service, and uses CPU computing power to calculate inference results.

[0040] In this embodiment, the heterogeneous scheduling service receives inference requests, splits the inference requests to obtain at least one inference task, and schedules the inference tasks to a long task queue or a short task queue according to the encoding length of each inference task.

[0041] In some embodiments, such as Figure 3 As shown, the heterogeneous computing power scheduling system is configured with a batch encoder, a task cache, a scheduler, a hardware performance fingerprint database, and a batch decoder. When the heterogeneous scheduling service receives an inference request sent by the user, the inference request first enters the batch encoder for preprocessing. The processed inference request is then scheduled to the task cache. The scheduler, based on the inference performance data stored in the hardware performance fingerprint database, schedules the inference tasks in the task cache to the GPU inference service or the CPU inference service.

[0042] In some embodiments, the method described above, which splits the inference request to obtain at least one inference task and schedules the inference tasks to a long task queue or a short task queue according to the encoding length of each inference task, is as follows: Figure 4As shown, it includes the following steps:

[0043] Step 1011: Divide the inference request into at least one inference task based on the prompt words included in the inference request, and bind the at least one inference task to the unique request identifier of the inference request. The number of at least one inference task is the same as the number of prompt words included in the inference request.

[0044] It should be noted that a single inference request may include one prompt or multiple prompts. For example, the prompts for an inference request may be, "Introduce the long prompt of xxx", "Introduce the short prompt of xxx", "Introduce the CPU", "hello world!", "ni hao!".

[0045] In some embodiments, when a single inference request includes multiple prompts, the batch encoder splits the multiple prompts of the single inference request into multiple inference tasks, and binds the multiple inference tasks to the request ID of the inference request. When a single inference request includes only one prompt, it is not split and is converted into a single inference task.

[0046] Step 1012: Encode each reasoning task in at least one reasoning task into a word sequence and obtain the sequence length of the word sequence.

[0047] In some embodiments, the batch encoder uses the tokenizer of the inference model to encode each inference task (i.e., prompt), converting the text into a sequence of tokens, and then calculating the length of the token sequence.

[0048] Step 1013: Schedule inference tasks with sequence lengths greater than the long-short classification threshold to the long task queue, and schedule inference tasks with sequence lengths less than or equal to the long-short classification threshold to the short task queue; wherein, the long-short classification threshold is obtained by the system during the initialization phase based on user service-level target indicators and the second inference performance data of the second processor inference service.

[0049] In some embodiments, during the initialization phase of the heterogeneous computing power scheduling system, long and short classification thresholds are calculated and stored based on user service-level target metrics and inference performance data of the second processor inference service.

[0050] In some embodiments, the Service Level Objective (SLO) metric includes the maximum latency of the first metaword. (seconds) and the maximum interval between generating each lexical unit (seconds), the SLO metric data is specified by the user when the inference system starts.

[0051] In some embodiments, the method further includes: obtaining first processor information corresponding to the first processor inference service and second processor information corresponding to the second processor inference service; based on the first processor information and the second processor information, determining whether the first inference performance data corresponding to the first processor inference service and the second inference performance data corresponding to the second processor inference service are stored in the first storage structure; if not, configuring the IP and port of the second processor inference service and the IP and port of the first processor inference service; calling the OpenAIAPI interface of the second processor inference service and the first processor inference service to perform inference performance testing, and obtaining the first inference performance data and the second inference performance data; and storing the first inference performance data and the second inference performance data into the first storage structure.

[0052] In some embodiments, the first storage structure is a hardware performance fingerprint database, which records inference performance data for different CPU models (e.g., Intel Xeon vs. AMD EPYC) and GPU models (e.g., NVIDIA A100 vs. V100). The sources of this inference performance data are divided into two parts: one part is built-in data, which is imported after pre-testing by the inference system developers; the other part is data obtained after the CPU / GPU performance test process is triggered when the current inference system's GPU / CPU model is not in the built-in data, and then imported.

[0053] In some embodiments, the CPU / GPU performance testing process is as follows: Figure 4 As shown, initialization is performed first, configuring the IP and port of the CPU inference service and the GPU inference service. Then, streaming requests are constructed to call the OpenAI API interfaces of the CPU inference service and the GPU inference service respectively to perform inference performance testing. The test code records the arrival time of the first token (TTFT) and the arrival time of each subsequent token, and calculates the average time interval (TPOT). After the test is completed, the prefill performance and decode performance are calculated based on the input data, returned data and recorded time points.

[0054] In some embodiments, inference performance data includes prefill performance and decoding performance, etc. Prefill performance includes the number of concurrent connections corresponding to different input lengths, which corresponds to the first-to-first-term latency (TTFT) and throughput. In one example, the storage structure for the number of concurrent connections corresponding to different input lengths of a certain CPU model is: Input length → {Concurrency: (TTFT, Throughput)…}, such as including:

[0055] 128: {1: (0.08, 1600), 4: (0.12, 1067), …},

[0056] 512: {1: (0.25, 2048), 4: (0.38, 1347), …},

[0057] 1024: {1: (0.48, 2133), 4: (0.72, 1422), …}

[0058] like The maximum input length among the two data sets above that meet the requirement of 0.26 is 512.

[0059] In one example, Prefill performance The calculation formula is as follows:

[0060]

[0061] in, The number of input prompt words is TTFT, where TTFT is the first-word delay.

[0062] In one example, Decode performance The calculation formula is as follows:

[0063]

[0064] in, The average time interval is given, and the performance data is measured in tokens per second.

[0065] In some embodiments, the method for obtaining the long-short classification threshold includes: obtaining the maximum input length that satisfies the maximum delay of the first word in the second inference performance data, and using the maximum input length as the long-short classification threshold.

[0066] In one example, the long / short classification threshold It is calculated using the following formula:

[0067]

[0068] in, This indicates that the inference performance data corresponding to the CPU inference service meets the following criteria. Maximum input length under the given conditions.

[0069] Since the inference performance data corresponding to the CPU inference service is stored in the hardware performance fingerprint database, the long / short classification threshold can also be expressed as the threshold for the inference performance data of the corresponding CPU inference service in the hardware performance fingerprint database that satisfies the following conditions. The maximum input length under certain conditions. This can be understood as follows: different CPU models correspond to different inference performance data for their CPU inference services, and the inference performance data for each CPU model includes multiple input lengths. The maximum input length that satisfies the given conditions is selected from these multiple input lengths. Given one or more initial input lengths under certain conditions, obtain the maximum input length from the initial input lengths.

[0070] This embodiment uses a hardware performance fingerprint database to store inference performance data, which can reduce the initial data detection time of the scheduling system and enable the scheduling service to start quickly.

[0071] In some embodiments, the batch encoder obtains the long-short classification threshold and compares the sequence length of the word sequence of each inference task with the long-short classification threshold. Inference tasks with a sequence length greater than the long-short classification threshold are scheduled to the long task queue, and inference tasks with a sequence length less than or equal to the long-short classification threshold are scheduled to the short task queue.

[0072] In some embodiments, a task cache is used to cache inference tasks, comprising a long task queue and a short task queue, following a first-in, first-out (FIFO) principle. The batch encoder inserts inference tasks of varying lengths into the long and short task queues.

[0073] Step 1014: Update the task status of the reasoning task to pending.

[0074] In some embodiments, the batch encoder inserts inference tasks of different lengths into long task queues and short task queues and marks the task status as pending.

[0075] Traditional inference task types have poor adaptability. Different inference tasks (long input / short input, long output / short output, etc.) are not as compatible with different processors. Traditional fixed-ratio scheduling strategies cannot allocate tasks according to the type of inference task.

[0076] This embodiment considers that the second processor's inference service has short input affinity, making it more suitable for handling inference tasks with short inputs. Based on the user-defined Service Level Objective (SLO) metric, a threshold for the length of inference tasks is calculated. Inference tasks of different lengths are scheduled to long task queues and short task queues according to this threshold. Inference tasks are categorized based on their length type to achieve inference task type adaptability. This embodiment splits and encodes single inference requests for multiple inference tasks, groups inference tasks with similar input lengths, increases the GPU / CPU concurrent inference computation time, and achieves reasonable long / short inference task scheduling, fully utilizing the computing performance of GPU / CPU.

[0077] Step 102: When it is detected that there are pending inference tasks in the two task queues, the working status of the first processor inference service is determined based on the first number of tasks and the maximum concurrency of the first processor. The maximum concurrency of the first processor is obtained based on the current conditions of the two task queues. The first number of tasks is the number of tasks being processed by the first processor inference service in the two task queues.

[0078] In this embodiment, the scheduling principle for inference tasks includes: if the first processor resource is idle, it is prioritized for scheduling to the first processor inference service; the second processor inference service has short input affinity and is more suitable for processing inference tasks in the short task queue. Therefore, this embodiment first obtains the working state of the first processor inference service, which includes an idle state and a non-idle state.

[0079] In some embodiments, the heterogeneous scheduling service monitors in real time whether there are any inference tasks with pending status in the long task queue and the short task queue. If there are pending inference tasks, scheduling is performed.

[0080] In some embodiments, the current condition of the two task queues includes the first average length of the input lexical units of the current inference task in the two task queues.

[0081] In some embodiments, the method described above for determining the working status of the first processor inference service based on the first number of tasks and the maximum concurrency of the first processor is as follows: Figure 5 As shown, it includes the following steps:

[0082] Step 1021: Obtain the first average length of the input lexical units of the current inference task in the two task queues and the first inference performance data corresponding to the first processor inference service.

[0083] In this embodiment, the inference tasks in the long task queue and the short task queue have multiple task states, including pending, being processed by the first processor, being processed by the second processor, and completed. The number of inference tasks in both task queues marked as being processed by the first processor is obtained, which is the first task count, also known as the number of tasks being processed by the first processor. .

[0084] In this embodiment, the first processor inference service can process inference tasks in the long task queue and the short task queue. Therefore, this step obtains the first average length of the input lexical units of the current inference task in the two task queues.

[0085] Step 1022: Based on the user service-level target metrics, the first inference performance data, and the first average length, obtain the maximum concurrency of the first processor.

[0086] In some embodiments, the maximum concurrency of the first processor is obtained using the following formula, based on user service-level target metrics, first inference performance data, and a first average length. :

[0087]

[0088] in: Indicates the maximum delay of the first word. This represents the maximum interval between each generated lexical unit. This indicates that the inference performance data corresponding to the first processor inference service meets the following requirements. Maximum pre-filled throughput under the given conditions; , represents the first average length of the input lexical units for the reasoning task in the two task queues; This indicates that the first inference performance data corresponding to the first processor inference service satisfies... Maximum concurrency under the given conditions; and For safety, the default value is 0.95 to prevent overload.

[0089] This embodiment designs a dynamic formula for calculating the maximum concurrency of the first processor. Based on the inference tasks in the current long / short task queues, it dynamically calculates the maximum concurrency of the first processor under the current inference task conditions. This solves the problem that a fixed concurrency cannot cope with multiple concurrent inference tasks, and can maximize the concurrency of the first processor to improve inference performance.

[0090] Step 1023: Determine whether the maximum concurrency of the first processor is greater than the number of the first tasks. If it is greater, determine that the working state of the inference service of the first processor is idle.

[0091] In this embodiment, if the first processor has the maximum concurrency Greater than the number of the first task If the condition is met, the first processor inference service is considered to be in an idle state; otherwise, the first processor inference service is considered to be in a non-idle state.

[0092] This embodiment eliminates the need to monitor processor utilization and inference service concurrency. While scheduling inference tasks, it synchronously retrieves the maximum concurrency of the first processor at the current moment from the task queue, avoiding scheduling errors caused by a mismatch between performance monitoring and scheduling decisions. Furthermore, based on the tasks in the current long / short task queues, it dynamically calculates the maximum concurrency of the first processor under the current inference task conditions. This solves the problem that a fixed concurrency level cannot handle multiple concurrent inference tasks, maximizing processor concurrency and thus maximizing inference performance.

[0093] Step 103: Based on the working status of the first processor inference service, schedule the inference task to be processed to the first processor inference service or the second processor inference service.

[0094] In some embodiments, the above-described method for scheduling pending inference tasks to the first processor inference service or the second processor inference service based on the working state of the first processor inference service is as follows: Figure 6 As shown, it includes the following steps:

[0095] Step 1031: If the working state of the first processor inference service is idle, obtain the first difference M between the maximum concurrency of the first processor and the number of first tasks.

[0096] In this embodiment, the first difference M = In order to obtain from the long task queue and the short task queue Each pending inference task is scheduled to the first processor inference service.

[0097] Step 1032: If the inference tasks to be processed exist in two task queues respectively, and the number of inference tasks to be processed in the long task queue is greater than or equal to M, retrieve M inference tasks to be processed from the long task queue and schedule them to the first processor inference service, and update the task status of the M inference tasks to be processed.

[0098] In this embodiment, another scheduling principle for the first processor inference service is: the scheduling priority is long task queue tasks > short task queue tasks, that is, the first processor inference service prioritizes long inference task types.

[0099] Step 1033: If the number of pending inference tasks in the long task queue is less than M, obtain M pending inference tasks from the two task queues and schedule them to the first processor inference service. The M pending inference tasks include all pending inference tasks in the long task queue, and update the task status of the M inference tasks to "processing".

[0100] If the number of pending reasoning tasks in the long task queue is less than Then, all pending inference tasks and a portion of the inference tasks in the short task queue are retrieved from the long task queue, resulting in M ​​pending inference tasks, which are then scheduled to the first processor inference service.

[0101] Step 1034: If the working state of the first processor inference service is non-idle, determine whether the inference task to be processed is located in the short task queue.

[0102] Because the second processor inference service has short input affinity, it is more suitable for processing inference tasks in the short task queue. When the first processor inference service is idle, it is also necessary to determine whether the inference task to be processed is a short inference task.

[0103] Step 1035: If yes, obtain the second task number, the second average length of the input lexical units of the current inference task in the short task queue, and the second inference performance data, wherein the second task number is the number of tasks processed by the second processor inference service in the short task queue.

[0104] If the inference task to be processed is a short inference task, it is necessary to determine whether the current state of the second processor can handle the inference task. To determine whether the current state of the second processor can handle the inference task, it is necessary to obtain the current maximum concurrency of the second processor. Therefore, it is necessary to obtain the second average length of the input tokens of the current inference task in the short task queue and the second inference performance data.

[0105] Step 1036: Based on the user service-level target metrics, the second inference performance data, and the second average length, the maximum concurrency of the second processor is obtained.

[0106] In some embodiments, the maximum concurrency of the second processor is obtained by the following formula. :

[0107]

[0108] in: Indicates the maximum delay of the first word. This represents the maximum interval between each generated lexical unit. This indicates that the inference performance data corresponding to the second processor inference service meets the requirements. Maximum pre-filled throughput under the given conditions; , represents the second average length of the input lexical units for the reasoning task in the short task queue; This indicates that the inference performance data corresponding to the second processor inference service meets the requirements. Maximum concurrency under the given conditions; and This is for the safety factor.

[0109] This embodiment designs a dynamic formula for calculating the maximum concurrency of the second processor. Based on the inference tasks in the current short task queue, it dynamically calculates the maximum concurrency of the second processor under the current inference task conditions. This solves the problem that a fixed concurrency level cannot handle multiple concurrent inference tasks, and can maximize the concurrency of the second processor to improve inference performance. At the same time, it introduces scheduling decisions driven by the SLO metric, which enables the full utilization of the heterogeneous computing power of different processors and improves inference performance in resource-constrained environments without affecting the user's experience of inference performance.

[0110] Step 1037: If the number of second tasks is less than the maximum concurrency of the second processor, obtain the second difference N between the maximum concurrency of the second processor and the number of second tasks, schedule the N pending inference tasks in the short task queue to the inference service of the second processor, and update the task status of the N pending inference tasks to "processing".

[0111] In this embodiment, the affinity between the first processor and the second processor varies depending on the length of the inference task. When the load on the first processor increases, more short inference tasks that are compatible with the second processor are scheduled to the second processor, thereby reducing the difference in inference performance between the first processor and the second processor.

[0112] Step 1038: Retrieve the inference tasks with the task status of "completed" from the two task queues, and delete the completed inference tasks from the two task queues.

[0113] In some embodiments, the decoder retrieves inference tasks with a completed task status from the long task queue and the short task queue, and deletes inference tasks with a completed task status from the long task queue and the short task queue.

[0114] Step 1039: Decode the reasoning results corresponding to the completed reasoning tasks to obtain the decoded reasoning results; assemble the decoded reasoning results belonging to the same unique request identifier into batch processing results, and return the batch processing results to the user terminal.

[0115] In some embodiments, the decoder uses the tokenizer of the inference model to decode the inference results of the completed inference task, converts the word data into text data, and then assembles the inference results with the same request ID into a batch result and returns it to the user.

[0116] The heterogeneous computing power scheduling method in this application splits inference requests into inference task types of different lengths, implementing an affinity mechanism between long and short inference task types and different processors. Inference tasks with different processor affinities are prioritized for scheduling to the corresponding processor's inference service, reducing the performance differences between different processors. By introducing long / short task queues, different types of inference tasks are more rationally scheduled to inference services on different processors, maximizing the utilization of the hybrid computing power of heterogeneous processors and improving the overall throughput of the inference service. Simultaneously, it eliminates the need to monitor processor utilization and inference service concurrency; the concurrency status of the processor's inference service at the current moment can be synchronously obtained from the task queue during inference task scheduling, avoiding erroneous scheduling problems caused by the asynchrony between performance monitoring and scheduling decisions. Based on the tasks in the current long / short task queues, the maximum processor concurrency under the current inference task conditions is dynamically calculated, solving the problem that a fixed concurrency level cannot handle multiple concurrent inference tasks, maximizing the processor's concurrency and improving inference performance. A dynamic scheduling process is implemented, solving the problem that fixed-ratio allocation methods cannot adapt to inference scenarios with dynamically changing loads. By making full use of the hybrid computing resources of heterogeneous processors, the problem of the first processor of the AI ​​inference server running under high load while the second processor's resources are idle and wasted is solved.

[0117] To clearly illustrate the above embodiments, specific examples will now be used for explanation. Figure 3 This is a schematic diagram of the scheduling process for a heterogeneous scheduling service provided by an embodiment of this application. Figure 3 As shown, the heterogeneous inference computing power scheduling method of this application is applied to a heterogeneous computing power scheduling system, which is equipped with a batch encoder, a task cache, a scheduler, a hardware performance fingerprint library, and a batch decoder.

[0118] When the heterogeneous scheduling service starts, the scheduler first detects the GPU and CPU models in the current environment. Then, it retrieves the corresponding inference performance data from the built-in hardware performance fingerprint database. If no inference performance data for the corresponding GPU and CPU models is detected, the GPU / CPU performance test program is launched to perform inference performance tests on the GPU and CPU models in the current environment. After the test is completed, the obtained inference performance data is stored in the hardware fingerprint database. When the heterogeneous scheduling service receives an inference request from the user, the inference request first enters the batch encoder for preprocessing.

[0119] The processing flow of the batch encoder is as follows: Figure 7As shown, the prompt of the inference request is split, the tokenizer of the inference model is used to encode the prompt, the text is converted into a token, the token length is calculated, and the tokens are allocated according to the long and short classification thresholds calculated by the scheduler. They are then saved to the long task queue and the short task queue in the task cache, respectively, and each inference task is bound to the ID of the inference request.

[0120] The scheduler retrieves inference tasks to be processed from the long task queue and the short task queue and marks the task status as GPU processing. After the inference is completed, the inference result is saved to the corresponding task data table and the task status is updated to complete. The batch decoder retrieves tasks with the task status of completed from the long task queue and the short task queue and deletes the completed tasks from the long task queue and the short task queue.

[0121] The scheduler is the control center of the heterogeneous scheduling service. When the scheduling service starts, the scheduler obtains the GPU and CPU information of the current environment's hardware devices, then searches for the corresponding hardware device's inference performance data in the hardware performance fingerprint database. If the data is not found in the database, a performance test program is triggered to test the CPU and GPU inference performance, and the test results are saved to the hardware performance fingerprint database. Then, a scheduling policy is calculated based on the user-configured Service Level Acceptance (SLO) metrics. Finally, according to the scheduling policy, inference tasks are retrieved from the task cache and scheduled to the GPU / CPU inference service. The performance testing process for CPU and GPU inference performance is as follows: Figure 8 As shown, initialization is performed first, configuring the IP and port of the CPU inference service and the GPU inference service. Then, streaming request configuration is performed, calling the OpenAI API interfaces of the CPU inference service and the GPU inference service respectively, listening for streaming responses. The test code records the arrival time of the first token (TTFT) and the arrival time of each subsequent token, and calculates the average time interval (TPOT). After the test is completed, performance data is calculated based on the input data, returned data, and recorded time points.

[0122] The scheduler's scheduling logic is as follows: Figure 9 As shown, the scheduler monitors the two task queues in real time to see if there are any pending inference tasks. If so, it performs scheduling computation. First, it queries the number of tasks being processed by the GPU in both task queues. Then calculate the maximum GPU concurrency under the current task queue conditions. ,if Number less than This indicates that GPU resources are available at this time, so inference tasks are prioritized for scheduling to the GPU inference service, and the number of scheduled inference tasks is [number missing]. The scheduling priority is that inference tasks in the long task queue are higher than inference tasks in the short task queue, and the scheduling order is FIFO (First In, First Out). If Number greater than or equal to If the test GPU resources are not available, the inference task will be scheduled to the CPU inference service. First, it needs to be determined whether the inference task to be processed is in the short task queue. If so, it meets the task type requirements of the CPU inference service and is scheduled. Then, the number of tasks being processed by the CPU in the short task queue is queried. And calculate the maximum CPU concurrency under the current short task queue condition. ,like Less than The number of scheduled tasks is Each task is assigned to the CPU inference service, and the scheduling order is FIFO.

[0123] The batch decoder is used to process the inference tasks that have been completed. The processing flow is as follows: Figure 10 As shown, inference tasks with a completed status are retrieved from the two task queues of the task cache. At the same time as retrieving the completed inference task, the inference task is deleted from the task queue. Then, the inference result of the completed inference task is decoded using the tokenizer of the inference model, and the token data is converted into text data. Finally, the inference results with the same request ID are assembled into a batch result and returned to the user.

[0124] By rationally designing a heterogeneous computing power scheduling system composed of functional modules such as batch encoders, batch decoders, task caches, and schedulers, inference tasks are rationally allocated to maximize the utilization of GPU+CPU hybrid computing power and improve the overall throughput of inference services.

[0125] To implement the heterogeneous inference computing power scheduling method provided in this disclosure, this disclosure also provides a heterogeneous inference computing power scheduling device. The device is configured in a heterogeneous computing power scheduling system, which includes a first processor inference service and a second processor inference service. Upon startup, the system configures two task queues, including a long task queue and a short task queue. Figure 11 As shown, it includes:

[0126] The task partitioning module 1111 is used to receive inference requests, split the inference requests to obtain at least one inference task, and schedule the inference tasks to a long task queue or a short task queue according to the encoding length of each inference task.

[0127] The status acquisition module 1112 is used to determine the working status of the first processor inference service based on the first number of tasks and the maximum concurrency of the first processor when it is detected that there are pending inference tasks in the two task queues. The maximum concurrency of the first processor is obtained based on the current conditions of the two task queues. The first number of tasks is the number of tasks being processed by the first processor inference service in the two task queues.

[0128] The task scheduling module 1113 is used to schedule the inference tasks to be processed to the first processor inference service or the second processor inference service based on the working status of the first processor inference service.

[0129] Furthermore, in one possible implementation of this disclosure embodiment, the task partitioning module 1111 is specifically used for:

[0130] Based on the prompt words included in the inference request, the inference request is divided into at least one inference task, and at least one inference task is bound to the unique request identifier of the inference request. The number of at least one inference task is the same as the number of prompt words included in the inference request.

[0131] Encode each reasoning task in at least one reasoning task into a word sequence and obtain the sequence length of the word sequence;

[0132] Inference tasks with sequence lengths greater than the long-short classification threshold are scheduled to the long task queue, while inference tasks with sequence lengths less than or equal to the long-short classification threshold are scheduled to the short task queue. The long-short classification threshold is obtained during the system initialization phase based on user service-level target metrics and the second inference performance data of the second processor inference service.

[0133] Update the task status of the reasoning task to pending.

[0134] Furthermore, in one possible implementation of this disclosure embodiment, the current conditions of the two task queues include the first average length of the input lexical units of the current inference task in the two task queues, and the state acquisition module 1112 is specifically used for:

[0135] Obtain the first average length of the input lexical units of the current inference task in the two task queues and the first inference performance data corresponding to the first processor inference service;

[0136] Based on user service-level target metrics, first inference performance data, and first average length, the maximum concurrency of the first processor is obtained;

[0137] Determine whether the maximum concurrency of the first processor is greater than the number of the first tasks. If it is greater, determine whether the working state of the inference service of the first processor is idle.

[0138] Furthermore, in one possible implementation of this disclosure embodiment, the user service level target indicators include the maximum delay of the first word unit and the maximum interval between generating each word unit. When the state acquisition module 1112 obtains the maximum concurrency of the first processor based on the user service level target indicators, the first inference performance data, and the first average length, it is specifically used for:

[0139] Using the first formula, based on user service-level target metrics, first inference performance data, and first average length, the maximum concurrency of the first processor is obtained. The first formula is expressed as follows:

[0140]

[0141] in: Indicates the maximum delay of the first word. This represents the maximum interval between each generated lexical unit. This indicates that the first inference performance data corresponding to the first processor inference service satisfies... Maximum pre-filled throughput under the given conditions; , represents the first average length of the input lexical units for the reasoning task in the two task queues; This indicates that the first inference performance data corresponding to the first processor inference service satisfies... Maximum concurrency under the given conditions; and This is for the safety factor.

[0142] Furthermore, in one possible implementation of this disclosure embodiment, the user service-level target metric includes the maximum delay of the first word element. When the task partitioning module 1111 obtains the long / short classification threshold, it is specifically used for:

[0143] Obtain the maximum input length that satisfies the maximum delay of the first word in the second inference performance data, and use the maximum input length as the long / short classification threshold.

[0144] Furthermore, in one possible implementation of this disclosure embodiment, the device further includes a performance testing module 1114, used for:

[0145] Obtain the first processor information corresponding to the first processor inference service and the second processor information corresponding to the second processor inference service;

[0146] Based on the first processor information and the second processor information, determine whether the first storage structure stores the first inference performance data and the second inference performance data.

[0147] If not, configure the IP address and port of the second processor inference service and the IP address and port of the first processor inference service;

[0148] Call the OpenAI API interfaces of the second processor inference service and the first processor inference service to perform inference performance tests and obtain first inference performance data and second inference performance data;

[0149] Store the first inference performance data and the second inference performance data into the first storage structure.

[0150] Furthermore, in one possible implementation of this disclosure embodiment, the task scheduling module 1113 is specifically used for:

[0151] If the working state of the first processor inference service is idle, obtain the first difference M between the maximum concurrency of the first processor and the number of the first tasks;

[0152] If the inference tasks to be processed exist in two task queues respectively, and the number of inference tasks to be processed in the long task queue is greater than or equal to M, M inference tasks to be processed are obtained from the long task queue and scheduled to the first processor inference service, and the task status of the M inference tasks is updated to "processing".

[0153] If the number of pending inference tasks in the long task queue is less than M, M pending inference tasks are obtained from the two task queues and scheduled to the first processor inference service. These M pending inference tasks include all pending inference tasks in the long task queue, and the task status of these M inference tasks is updated to "processing".

[0154] Furthermore, in one possible implementation of this disclosure embodiment, the task scheduling module 1113 is further configured to:

[0155] If the working state of the first processor inference service is non-idle, determine whether the inference task to be processed is in the short task queue;

[0156] If so, obtain the second task count, which is the number of tasks processed by the second processor inference service in the short task queue;

[0157] Obtain the second average length and second inference performance data of the input lexical units for the current inference task in the short task queue;

[0158] Based on user service-level target metrics, second inference performance data, and second average length, the maximum concurrency of the second processor is obtained.

[0159] If the number of second tasks is less than the maximum concurrency of the second processor, obtain the second difference N between the maximum concurrency of the second processor and the number of second tasks;

[0160] Schedule N pending inference tasks from the short task queue to the second processor inference service, and update the task status of these N pending inference tasks to "processing".

[0161] Furthermore, in one possible implementation of this disclosure embodiment, the task scheduling module 1113 is further configured to:

[0162] Retrieve inference tasks with a completed status from both task queues and remove the completed inference tasks from both task queues;

[0163] The reasoning results corresponding to the completed reasoning tasks are decoded to obtain the decoded reasoning results. The decoded reasoning results belonging to the same unique request identifier are assembled into batch results and returned to the user.

[0164] It should be noted that the foregoing explanation of the heterogeneous inference computing power scheduling method embodiment also applies to the heterogeneous inference computing power scheduling device of this embodiment, and will not be repeated here.

[0165] Embodiments of this disclosure also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the heterogeneous inference computing power scheduling method.

[0166] Embodiments of this disclosure also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the heterogeneous inference computing power scheduling method when running.

[0167] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0168] The embodiments of this disclosure also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the heterogeneous inference computing power scheduling method.

[0169] Embodiments of this disclosure also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the heterogeneous inference computing power scheduling method.

[0170] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0171] The target detection method provided in this disclosure has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this disclosure without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this disclosure.

Claims

1. A heterogeneous inference computing power scheduling method, characterized in that, The method is applied to a heterogeneous computing power scheduling system, which includes a first processor inference service and a second processor inference service. The first processor inference service is a GPU inference service, and the second processor inference service is a CPU inference service. The system configures two task queues at startup, including a long task queue and a short task queue. The method includes: Receive an inference request, split the inference request to obtain at least one inference task; and schedule the inference task to the long task queue or the short task queue according to the encoding length of each inference task. When it is detected that there are pending inference tasks in the two task queues, the working status of the first processor inference service is determined based on the first number of tasks and the maximum concurrency of the first processor. The maximum concurrency of the first processor is obtained based on the current conditions of the two task queues. The first number of tasks is the number of tasks being processed by the first processor inference service in the two task queues. Based on the working status of the first processor inference service, the inference task to be processed is scheduled to the first processor inference service or the second processor inference service. The current conditions of the two task queues include the first average length of the input lexical units of the current inference task in the two task queues. The step of determining the working status of the first processor's inference service based on the first number of tasks and the maximum concurrency of the first processor includes: Obtain the first average length of the input lexical units of the current inference task in the two task queues and the first inference performance data corresponding to the first processor inference service; Based on the user service-level target metrics, the first inference performance data, and the first average length, the maximum concurrency of the first processor is obtained; Determine whether the maximum concurrency of the first processor is greater than the number of the first tasks. If it is greater, determine that the working state of the inference service of the first processor is idle.

2. The method according to claim 1, characterized in that, The inference request is split into at least one inference task; And according to the encoding length of each inference task, the inference tasks are scheduled to the long task queue or the short task queue, including: Based on the prompt word segmentation identifier included in the reasoning request, the reasoning request is divided into at least one reasoning task, and the at least one reasoning task is bound to the unique request identifier of the reasoning request, wherein the number of the at least one reasoning task is the same as the number of prompt words included in the reasoning request. Encode each of the at least one reasoning task into a word sequence, and obtain the sequence length of the word sequence; The inference tasks with sequence lengths greater than the long-short classification threshold are scheduled to the long task queue, and the inference tasks with sequence lengths less than or equal to the long-short classification threshold are scheduled to the short task queue. The long-short classification threshold is obtained during the system initialization phase based on user service-level target indicators and the second inference performance data of the second processor inference service. Update the task status of the inference task to pending.

3. The method according to claim 1, characterized in that, The user service level target metrics include the maximum delay of the first word element and the maximum interval between the generation of each word element. Based on the user service-level target metrics, the first inference performance data, and the first average length, the maximum concurrency of the first processor is obtained, including: Using the first formula, based on the user service-level target metric, the first inference performance data, and the first average length, the maximum concurrency of the first processor is obtained. The first formula is expressed as follows: in: This indicates the maximum delay of the first word. This indicates the maximum interval between each generated lexical unit. This indicates that the first inference performance data corresponding to the first processor inference service satisfies... Maximum pre-filled throughput under the given conditions; , represents the first average length of the input lexical units for reasoning tasks in the long task queue and the short task queue; This indicates that the inference performance data corresponding to the first processor inference service satisfies... Maximum concurrency under the given conditions; and For safety reasons, The number of input prompt tokens.

4. The method according to claim 2, characterized in that, The user service-level target metrics include the maximum delay of the first word element, and the method for obtaining the long / short classification threshold includes: Obtain the maximum input length that satisfies the maximum delay of the first word in the second inference performance data; The maximum input length is used as the long / short classification threshold.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the first processor information corresponding to the first processor inference service and the second processor information corresponding to the second processor inference service; Based on the first processor information and the second processor information, determine whether the first storage structure stores the first inference performance data and the second inference performance data. If not, configure the IP and port of the second processor inference service and the IP and port of the first processor inference service; Call the OpenAI API interfaces of the second processor inference service and the first processor inference service to perform inference performance testing and obtain the first inference performance data and the second inference performance data; The first inference performance data and the second inference performance data are stored in the first storage structure.

6. The method according to claim 1, characterized in that, The step of scheduling the pending inference task to the first processor inference service or the second processor inference service based on the working state of the first processor inference service includes: If the working state of the first processor inference service is idle, obtain the first difference M between the maximum concurrency of the first processor and the number of the first tasks; If the inference tasks to be processed exist in the two task queues respectively, and the number of inference tasks to be processed in the long task queue is greater than or equal to M, M inference tasks to be processed are obtained from the long task queue and scheduled to the first processor inference service, and the task status of the M inference tasks is updated to "processing". If the number of pending inference tasks in the long task queue is less than M, M pending inference tasks are obtained from the two task queues and scheduled to the first processor inference service. The M pending inference tasks include all pending inference tasks in the long task queue, and the task status of the M inference tasks is updated to "processing".

7. The method according to claim 1, characterized in that, The step of scheduling the pending inference task to the first processor inference service or the second processor inference service based on the working state of the first processor inference service includes: If the working state of the first processor inference service is non-idle, determine whether the inference task to be processed is located in the short task queue; If so, obtain the second task number, the second average length of the input lexical units of the current inference task in the short task queue, and the second inference performance data corresponding to the second processor inference service, wherein the second task number is the number of tasks processed by the second processor inference service in the short task queue; Based on user service-level target metrics, second inference performance data, and second average length, the maximum concurrency of the second processor is obtained; If the number of the second tasks is less than the maximum concurrency of the second processor, obtain the second difference N between the maximum concurrency of the second processor and the number of the second tasks; The N pending inference tasks in the short task queue are scheduled to the second processor inference service, and the task status of the N pending inference tasks is updated to "processing".

8. The method according to claim 2, characterized in that, The method further includes: Retrieve the reasoning tasks with the status of "completed" from the two task queues, and delete the completed reasoning tasks from the two task queues; The reasoning results corresponding to the completed reasoning tasks are decoded to obtain decoded reasoning results. The decoded reasoning results belonging to the same unique request identifier are assembled into batch processing results, and the batch processing results are returned to the user terminal.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program to implement the steps as described in any one of claims 1 to 8.

Images