Task scheduling methods, devices, computing equipment, storage media and program products

Abstract

This application provides a task scheduling method, apparatus, computing device, and computer-readable storage medium. The method includes: obtaining performance index values ​​and energy consumption index values ​​of each candidate accelerator for the target model to which the target task to be scheduled belongs; the candidate accelerators are at least two accelerators from all accelerators; calculating a suitability score for each candidate accelerator for the target task to be scheduled based on the performance index values ​​and energy consumption index values ​​of each candidate accelerator; and sending the target task to be scheduled to the target accelerator with the highest suitability score among the candidate accelerators. The solution of this application can select a more suitable target accelerator for task execution.

Description

Technical Field

[0001] This application relates to the field of heterogeneous computing device technology, and more specifically, to a task scheduling method, apparatus, computing device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the widespread application of deep learning technology, the requirements for model inference speed and energy efficiency are becoming increasingly stringent. Modern computing devices generally adopt heterogeneous architectures, integrating multiple accelerators, such as NPU (Neural Processing Unit), iGPU (Integrated Graphics Processing Unit), and GPGPU (General-Purpose Graphics Processing Unit), to provide the operating foundation for various models (such as image classification models, object detection models, and image segmentation models).

[0003] Currently, for heterogeneous computing devices, static scheduling strategies are typically used, employing fixed rules (such as "prioritizing NPUs") to schedule tasks to appropriate accelerators. However, this scheduling method is simplistic, and the accelerator assigned to a user's task is often not the most suitable, leading to low resource utilization. Summary of the Invention

[0004] The purpose of this application is to provide a task scheduling method, apparatus, computing device, computer-readable storage medium, and computer program product to solve the problems existing in the related art.

[0005] This application provides a task scheduling method applied to a CPU (Central Processing Unit) of a heterogeneous system, wherein the heterogeneous system further includes at least two accelerators connected to the CPU; the method includes: obtaining the performance index value and energy consumption index value of each candidate accelerator for the target model to which the target task to be scheduled belongs; the candidate accelerators are at least two accelerators among all accelerators; calculating the adaptation score of each candidate accelerator for the target task to be scheduled based on the performance index value and the energy consumption index value of each candidate accelerator; and sending the target task to be scheduled to the target accelerator with the highest adaptation score among the candidate accelerators.

[0006] In the above implementation scheme, by obtaining the performance and energy consumption values ​​of each candidate accelerator for the target model of the target task, and calculating the adaptation score of each candidate accelerator for the target task, the target task is sent to the target accelerator with the highest adaptation score for processing. In this way, during task scheduling, the adaptation of each candidate accelerator to the target task is comprehensively considered from both performance and energy consumption perspectives, making the target accelerator more suitable for the target task and maximizing the utilization of accelerator resources in the heterogeneous system, thus improving resource utilization.

[0007] Optionally, the performance metric values ​​include the inference latency parameter when the candidate accelerator runs the target model; obtaining the performance metric values ​​of each candidate accelerator for the target model to which the target scheduled task belongs includes: obtaining the inference latency (Inference_Latency) when each candidate accelerator runs the target model; for each candidate accelerator: determining the total inference latency of the candidate accelerator based on the Inference_Latency of the candidate accelerator and the existing tasks to be processed by the candidate accelerator; normalizing the total inference latency to obtain the inference latency parameter of the candidate accelerator.

[0008] In the above implementation, the total inference latency of a candidate accelerator is determined by combining the inference latency (Inference_Latency) of the candidate accelerator when running the target model with the existing tasks to be processed by the candidate accelerator. The resulting total inference latency for each candidate accelerator reflects the estimated total time required from the time the target task is assigned to each candidate accelerator until each candidate accelerator completes processing the target task. Therefore, the inference latency parameter obtained based on the total inference latency of each candidate accelerator, used as a performance indicator, can better reflect the performance of each candidate accelerator in processing the target task (the smaller the total inference latency, the better the performance), thereby improving the accuracy of the identified target accelerators. Furthermore, normalizing the total inference latency eliminates the influence of the parameter's unit, allowing for a more objective combination with the energy consumption indicator to calculate the fit score, making the resulting fit score more objective.

[0009] Optionally, the total inference latency of the candidate accelerator is determined based on its Inference_Latency and the number of pending tasks. This includes determining the total inference latency (Total_Latency) of the candidate accelerator according to the formula Total_Latency = Inference_Latency + Queue_Waiting_Time; where Queue_Waiting_Time is equal to the product of Inference_Latency and the number of pending tasks of the candidate accelerator.

[0010] In the above implementation, Total_Latency = Inference_Latency + Queue_Waiting_Time. Queue_Waiting_Time is equal to the product of Inference_Latency and the number of pending tasks already processed by the candidate accelerator. In other words, Queue_Waiting_Time represents the inference latency when the target model is run on the candidate accelerator. Inference_Latency represents the inference latency of each pending task already processed by the candidate accelerator, thus estimating the time required for the candidate accelerator to process all existing pending tasks. Therefore, by performing simple calculations based on only two data points—Inference_Latency and the number of pending tasks already processed by the candidate accelerator—Total_Latency can be quickly estimated, which improves the efficiency of task scheduling decisions (i.e., improves the efficiency of identifying the target accelerator), thereby improving task scheduling efficiency.

[0011] Optionally, the total inference latency is normalized to obtain the inference latency parameter of the candidate accelerator, including: normalizing the total inference latency Total_Latency according to the formula S_latency = (Latency_max - Total_Latency) / (Latency_max - Latency_min) to obtain the inference latency parameter S_latency of the candidate accelerator; wherein, Latency_max is the maximum total inference latency among the total inference latency of each candidate accelerator; and Latency_min is the minimum total inference latency among the total inference latency of each candidate accelerator.

[0012] In the above implementation, the maximum total inference delay (Latency_max) and the minimum total inference delay (Latency_min) among all candidate accelerators are normalized using the formula S_latency = (Latency_max - Total_Latency) / (Latency_max - Latency_min). This ensures that the obtained inference delay parameter S_latency is between 0 (inclusive) and 1 (inclusive), facilitating a more objective combination with energy consumption indicators to calculate the fit score, resulting in a more objective fit score. Furthermore, based on the inference delay parameter S_latency obtained from the above formula, the smaller the total inference delay of a candidate accelerator, the higher its inference delay parameter S_latency. Thus, the inference delay parameter S_latency of each candidate accelerator is positively correlated with its performance in processing the target scheduled task, thereby facilitating the determination of the fit score.

[0013] Optionally, the inference latency (Inference_Latency) of each candidate accelerator when running the target model is: an estimated value determined based on the historical inference latency recorded when the candidate accelerator has run the target model in the past; or, the pre-inference latency recorded when the candidate accelerator is used to perform pre-inference on the target model.

[0014] In the above implementation, an estimated value is determined based on the historical inference latency recorded when the candidate accelerator runs the target model in the past, and is used as the inference latency Inference_Latency when the candidate accelerator runs the target model. Alternatively, the pre-inference latency recorded when the candidate accelerator performs pre-inference on the target model is used as the inference latency Inference_Latency when the candidate accelerator runs the target model. The inference latency Inference_Latency obtained in this way is obtained after actual operation, and has higher reliability and accuracy. This makes the obtained inference latency parameter more reliable and accurate, and can improve the accuracy of the determined target accelerator.

[0015] Optionally, obtaining the energy consumption index value of each candidate accelerator for the target model to which the target scheduled task belongs includes: for each candidate accelerator: obtaining the energy consumption of the candidate accelerator when running the target model; normalizing the energy consumption to obtain the energy consumption index value of the candidate accelerator.

[0016] In the above implementation, the energy consumption of the candidate accelerators when running the target model is normalized to obtain the energy consumption index value of the candidate accelerators. Since the energy consumption of the candidate accelerators when running the target model can directly reflect the energy consumption of the candidate accelerators when executing the target scheduled task, using the normalized result as the energy consumption index value of the candidate accelerators can better reflect the energy consumption of each candidate accelerator when processing the target scheduled task, thereby improving the accuracy of the identified target accelerators. In addition, normalizing the energy consumption of the candidate accelerators when running the target model can eliminate the influence of the unit of the parameter itself, so as to combine it with the performance index value more objectively to calculate the adaptation score, making the obtained adaptation score more objective.

[0017] Optionally, the energy consumption of the candidate accelerator when running the target model is: an estimated value determined based on the historical energy consumption recorded when the candidate accelerator has run the target model in the past; or, the pre-inference energy consumption recorded when using the candidate accelerator to perform pre-inference on the target model.

[0018] In the above implementation, an estimated value is determined based on the historical energy consumption recorded when the candidate accelerator runs the target model in the past, or the pre-inference delay recorded when the candidate accelerator performs pre-inference on the target model is used as the pre-inference energy consumption when the candidate accelerator runs the target model. The energy consumption obtained in this way for calculating the energy consumption index value is obtained after actual operation, and has higher reliability and accuracy. This makes the obtained energy consumption index value more reliable and accurate, thereby improving the accuracy of the identified target accelerator.

[0019] Optionally, the energy consumption is normalized to obtain the energy consumption index value of the candidate accelerator, including: normalizing the energy consumption Energy according to the formula S_energy=(Energy_max - Energy) / (Energy_max – Energy_min) to obtain the energy consumption index value S_energy of the candidate accelerator; wherein, Energy_max is the maximum energy consumption among the energy consumption of each candidate accelerator; and Energy_min is the minimum energy consumption among the energy consumption of each candidate accelerator.

[0020] In the above implementation, the energy consumption of all candidate accelerators is normalized according to the formula S_energy = (Energy_max - Energy) / (Energy_max – Energy_min). This ensures that the obtained energy consumption index value S_energy is between 0 and 1, which facilitates a more objective calculation of the adaptation score by combining it with the performance index value. Furthermore, based on the energy consumption index value S_energy obtained from the above formula, the lower the energy consumption of a candidate accelerator, the higher its energy consumption index value S_energy. Thus, the energy consumption index value S_energy of each candidate accelerator is positively correlated with its energy consumption performance when processing the target scheduled task, thereby facilitating the determination of the adaptation score.

[0021] Optionally, an adaptation score for each candidate accelerator to the target scheduled task is calculated based on the performance index value and energy consumption index value of each candidate accelerator, including: obtaining the current operating mode of the heterogeneous system; obtaining a first weighting coefficient and a second weighting coefficient corresponding to the current operating mode; and for each candidate accelerator: using the first weighting coefficient and the second weighting coefficient to perform a weighted summation of the performance index value and the energy consumption index value to obtain the adaptation score of the candidate accelerator to the target scheduled task.

[0022] In the above implementation, by obtaining the first and second weighting coefficients corresponding to the current operating mode of the heterogeneous system, the performance index values ​​and energy consumption index values ​​are weighted and summed to obtain the adaptation score of each candidate accelerator for the target scheduling task. In this way, different first and second weighting coefficients can be set for different operating modes, thereby enabling preferential selection for different operating modes, making the selected target accelerators more suitable for the current operating mode requirements of the heterogeneous system.

[0023] Optionally, an adaptation score for each candidate accelerator to the target scheduled task is calculated based on the performance index value and energy consumption index value of each candidate accelerator, including: obtaining the processing mode of the heterogeneous system for the target scheduled task; obtaining the first weight coefficient and the second weight coefficient corresponding to the processing mode; and for each candidate accelerator: using the first weight coefficient and the second weight coefficient to perform a weighted summation of the performance index value and the energy consumption index value to obtain the adaptation score of the candidate accelerator to the target scheduled task.

[0024] In the above implementation, by obtaining the first and second weighting coefficients corresponding to the processing modes of the heterogeneous system for the target task to be scheduled, and then weighting and summing the performance index values ​​and energy consumption index values, the suitability score of each candidate accelerator for the target task to be scheduled is obtained. In this way, different processing modes can be set for different tasks to be scheduled, and different first and second weighting coefficients can be set for different processing modes. This allows for a preference for different processing modes, making the selected target accelerators more suitable for the processing modes of the heterogeneous system for the target task to be scheduled.

[0025] Optionally, before obtaining the performance index and energy consumption index values ​​of each candidate accelerator for the target model to which the target scheduled task belongs, the method further includes: determining whether a preset batch processing condition is triggered; if the batch processing condition is not triggered, then determining the currently scheduled task in the preset total task queue as the target scheduled task; the currently scheduled task is: the task currently scheduled in the total task queue when scheduled according to a preset sequential scheduling order; the target model is the model to which the currently scheduled task belongs; if the batch processing condition is triggered, then determining the N batch-processable tasks in the total task queue as the target scheduled task; the target scheduled task includes the currently scheduled task.

[0026] In the above implementation, by setting batch processing conditions, when the batch processing conditions are not triggered, the target accelerators are confirmed and tasks are scheduled one by one for the currently scheduled tasks in the total task queue. When the batch processing conditions are triggered, a target accelerator is simultaneously determined for the N tasks belonging to the target model in the total task queue and tasks are scheduled. In this way, when the batch processing conditions are not triggered (for example, when there are few tasks to be scheduled), confirming the target accelerators one by one and scheduling tasks can maximize the utilization of the target accelerators for task processing, thereby improving task processing efficiency. When there are many tasks to be scheduled, batch processing can improve the overall task processing efficiency, thereby improving the overall resource utilization of the heterogeneous system.

[0027] Optionally, the batch processing conditions include at least one of the following: the minimum total inference latency (Latency_min) among the total inference latency of each accelerator running the target model is higher than a preset first threshold; the minimum total inference latency (Latency_min) among the total inference latency of each accelerator running the target model is higher than the preset first threshold, and the number of batchable tasks in the total task queue after a preset first time interval is greater than a preset second threshold; the number of batchable tasks in the total task queue is greater than a preset third threshold; and the second threshold and the third threshold are greater than or equal to the minimum value among the batch values ​​supported by each accelerator.

[0028] In the above implementation, when the minimum total inference latency (Latency_min) among the total inference latency of each accelerator running the target model is higher than a preset first threshold, it can be considered that there is a large backlog of tasks to be processed in each accelerator. This can be used as a batch processing condition to achieve more accurate and fine-grained dynamic batch processing control. When the number of batch-processable tasks in the total task queue is greater than a preset third threshold, it can be intuitively considered that there is a large backlog of tasks to be scheduled in the total task queue. This can also be used as a batch processing condition to achieve more accurate and fine-grained dynamic batch processing control. Furthermore, by using the condition that the minimum total inference latency (Latency_min) among the total inference latency of each accelerator running the target model is higher than the preset first threshold, and the number of batch-processable tasks in the total task queue is greater than the preset second threshold after a preset first time interval, the backlog of tasks in each dedicated processor and the backlog of tasks to be scheduled in the total task queue are considered simultaneously, thereby achieving more accurate and fine-grained dynamic batch processing control.

[0029] Optionally, when the batch processing condition is triggered, the candidate accelerator is: among all accelerators, the accelerator that supports a batch value less than or equal to the number of tasks that can be batch-processed in the total task queue, and meets the preset accelerator participation scheduling conditions.

[0030] In the above implementation, when the batch processing condition is triggered, the accelerators that support a batch value less than or equal to the number of batch-processable tasks in the total task queue and meet the preset accelerator participation scheduling conditions are selected as candidate accelerators. In this way, the selected candidate accelerators can meet the requirements of the number of batch-processing scheduling tasks when performing batch processing, so that the selected target accelerators can perform batch processing.

[0031] Optionally, the tasks that can be batch-processed in the total task queue are those belonging to the target model.

[0032] In the above implementation, tasks belonging to the target model are batch-processed as batch-processable tasks. After task allocation, when the target accelerator executes batch-processed tasks, it only needs to load one target model, which can better utilize the performance of the target accelerator.

[0033] Optionally, if the batch processing condition is not triggered, before obtaining the performance index value and energy consumption index value of each candidate accelerator for the target model to which the target scheduled task belongs, the method further includes: determining all accelerators among the accelerators that meet the preset accelerator participation scheduling conditions as the candidate accelerators.

[0034] In the above implementation, by selecting all accelerators that meet the preset accelerator participation scheduling conditions as candidate accelerators, the risk of incorrectly assigning the target scheduled task to an accelerator that does not meet the accelerator participation scheduling conditions, resulting in the target scheduled task being unable to be processed normally, can be avoided.

[0035] Optionally, after sending the target task to be scheduled to the target accelerator with the highest adaptation score among the candidate accelerators, the method further includes: in response to a preset task rescheduling condition being triggered, re-determining new candidate accelerators from among the accelerators; wherein the new candidate accelerators do not include the target accelerator; obtaining the performance index value and energy consumption index value of each of the new candidate accelerators for the target model; calculating the adaptation score of each of the new candidate accelerators for the target task to be scheduled based on the performance index value and energy consumption index value of each of the new candidate accelerators; and re-sending the target task to be scheduled to the accelerator with the highest adaptation score among the new candidate accelerators.

[0036] In the above implementation, by setting task rescheduling conditions, a new target accelerator can be selected when the task rescheduling conditions are triggered (e.g., when the target accelerator fails). This allows the target scheduled tasks to be reasonably redistributed to the new accelerator for execution, increasing the probability that the target scheduled tasks can be executed correctly and improving the reliability of the heterogeneous system.

[0037] Optionally, the task rescheduling conditions include at least one of the following: the target accelerator malfunctions; the target accelerator processes a single task for more than a preset second duration.

[0038] In the above implementation, if the target accelerator fails or the time taken for the target accelerator to process a single task exceeds a preset second time limit, the target accelerator can be considered to have lost its ability to execute the target scheduled task at least for the time being. In this case, a new target accelerator is selected and the target scheduled task is reassigned to the new target accelerator, which can increase the probability that the target scheduled task can be executed correctly and improve the reliability of the heterogeneous system.

[0039] This application embodiment also provides a task scheduling device applied to a central processing unit (CPU) of a heterogeneous system, wherein the heterogeneous system further includes at least two types of accelerators connected to the CPU; the task scheduling device includes: an acquisition module, used to acquire the performance index value and energy consumption index value of each candidate accelerator for the target model to which the target task to be scheduled belongs; the candidate accelerators are at least two accelerators among all accelerators; a calculation module, used to calculate the adaptation score of each candidate accelerator for the target task to be scheduled based on the performance index value and the energy consumption index value of each candidate accelerator; and a sending module, used to send the target task to be scheduled to the target accelerator with the highest adaptation score among the candidate accelerators.

[0040] This application also provides a computing device, a heterogeneous system, and a memory. The heterogeneous system includes a CPU and at least two accelerators connected to the CPU. The CPU is used to execute one or more programs stored in the memory to implement any of the above-described task scheduling methods.

[0041] This application also provides a computer-readable storage medium storing one or more programs that can be executed by a CPU to implement any of the above-described task scheduling methods.

[0042] This application also provides a computer program product, which includes a computer program. When the computer program is run by the CPU, the CPU executes any of the above-described task scheduling methods. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A schematic diagram of the basic structure of an exemplary heterogeneous system provided in this application embodiment; Figure 2 A flowchart illustrating a task scheduling method provided in an embodiment of this application; Figure 3 A schematic diagram illustrating a process for obtaining inference delay parameters provided in an embodiment of this application; Figure 4 This application provides a schematic diagram of a process for obtaining energy consumption index values ​​in an embodiment of the present application. Figure 5A schematic diagram of a software architecture provided for an embodiment of this application; Figure 6 This is a schematic diagram of the basic structure of a task scheduling device provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0046] To facilitate understanding of the solutions and advantages of the embodiments of this application, some relevant technical terms that may be used will be explained below: Accelerator: In this application, it refers to a device that can be used to provide the running foundation for various models (such as image classification models, object detection models, image segmentation models, etc.), including but not limited to GPU (Graphics Processing Unit), NPU, iGPU, GPGPU and other processors.

[0047] Heterogeneous system: In this application, it refers to a system consisting of a CPU and at least two types of accelerators, where the CPU schedules tasks to the accelerators. For example... Figure 1 As shown, Figure 1 The basic architecture of a possible heterogeneous system is shown, including a CPU and at least two accelerators ( Figure 1 It shows NPU, iGPU, GPGPU), and CPU connected to at least two of these accelerators to enable task scheduling to specific accelerators.

[0048] In this embodiment of the application, task-to-accelerator scheduling refers to: assigning a task to a specific accelerator, and during execution, the CPU sends the task to the specific accelerator, thereby completing task scheduling.

[0049] Accelerators to be selected: In this application, accelerators that participate in the scheduling of this mission are selected.

[0050] Performance metrics of candidate accelerators: In this application, these metrics refer to those that reflect the performance of candidate accelerators, such as those that reflect the inference latency of candidate accelerators when processing tasks.

[0051] Energy consumption index of the candidate accelerator: In this application, it refers to the index value that reflects the energy consumption of the candidate accelerator.

[0052] Computing device: In this application, it refers to an electronic device with a heterogeneous system that can deploy models and perform model tasks.

[0053] Target task to be scheduled: In this application, it refers to the model task that needs to be scheduled this time.

[0054] To improve scheduling efficiency and resource utilization in heterogeneous systems, please refer to... Figure 2 As shown, Figure 2 This is a basic flowchart illustrating a task scheduling method applicable to CPUs in heterogeneous systems, as provided in this application embodiment, including: S201: Obtain the performance index and energy consumption index of each candidate accelerator for the target model of the target task to be scheduled.

[0055] In one optional embodiment of this application, all accelerators in the heterogeneous system can be used as candidate accelerators.

[0056] In another optional embodiment of this application, accelerator participation scheduling conditions can be set, and all accelerators in the heterogeneous system that meet the accelerator participation scheduling conditions can be selected as candidate accelerators, thereby eliminating some accelerators that obviously do not meet the requirements and improving task scheduling efficiency.

[0057] For example, the conditions for an accelerator to participate in scheduling may include at least one of the following: accelerator failure; the time it takes for an accelerator to process a single task exceeds a preset warning time; the number of tasks to be processed in the accelerator exceeds a preset warning number threshold, etc.

[0058] It is understandable that in the event of an accelerator malfunction, the accelerator is not processing tasks normally, and the time it takes for the accelerator to process a single task exceeds the preset warning time, or the number of tasks to be processed in the accelerator exceeds the preset warning number threshold, all of which mean that the current workload of the accelerator is relatively saturated, or even overloaded. Therefore, using at least one of the above conditions as the accelerator participation in scheduling can more accurately eliminate unsuitable accelerators and improve task scheduling efficiency.

[0059] In this embodiment, the preset alert duration and preset alert quantity threshold can be set by the engineer according to actual needs.

[0060] For example, suppose Figure 1 If the iGPU among the NPU, iGPU, and GPGPU connected to the CPU does not meet the conditions for accelerator participation in scheduling (e.g., a malfunction), then the candidate accelerators that can participate in the scheduling of this task are the NPU and GPGPU.

[0061] In this embodiment of the application, when setting the conditions for accelerator participation in scheduling, if there is only one candidate accelerator, it can be directly used as the target accelerator, and the target task to be scheduled can be sent to that candidate accelerator. For example, suppose... Figure 1If neither the NPU nor the iGPU meets the conditions for accelerator participation in scheduling among the NPU, iGPU, and GPGPU connected to the CPU, then the only candidate accelerator that can participate in the scheduling of this task is the GPGPU. In this case, the subsequent steps can be skipped, and the target task to be scheduled can be directly sent to the GPGPU.

[0062] In this embodiment, the target model is the model to which the target task to be scheduled belongs. For example, if the target task to be scheduled is an image classification model task, then the target model is an image classification model.

[0063] In an optional embodiment of this application, the performance metric value may include an inference latency parameter when the selected accelerator runs the target model. The inference latency parameter may be based on the following... Figure 3 Calculated as shown: S301: Obtain the inference latency (Inference_Latency) when each candidate accelerator runs the target model.

[0064] In an optional embodiment of this application, the inference latency (Inference_Latency) of any candidate accelerator running the target model can be an estimated value determined based on historical inference latencies recorded when the candidate accelerator has historically run the target model. For example, a heterogeneous system can continuously record the inference latency of each candidate accelerator running the target model during operation. These inference latencies are historical inference latencies. When S301 needs to be executed, these historical inference latencies can be obtained, or the average value of historical inference latencies within the most recent preset time period or a preset number of times can be obtained to obtain the estimated inference latency (Inference_Latency) of each candidate accelerator running the target model.

[0065] In another optional embodiment of this application, the inference latency (Inference_Latency) when any candidate accelerator runs the target model can also be the pre-inference latency recorded when the candidate accelerator performs pre-inference on the target model. For example, when the heterogeneous system has just started or the target model has just been loaded onto the candidate accelerator, the candidate accelerator can run the target model for one round of tasks, and the inference latency (i.e., pre-inference latency) at this time can be recorded. This pre-inference latency can then be used as the inference latency (Inference_Latency) when the candidate accelerator runs the target model for subsequent computation.

[0066] It is understood that the two optional implementation methods described above can be used simultaneously. When used simultaneously, the pre-inference latency in the second optional implementation method can be used as the initial Inference_Latency, and the estimated value in the first optional implementation method can be used as the Inference_Latency after the target model is actually run on the candidate accelerator.

[0067] In another optional implementation of the embodiments of this application, engineers may collect the inference latency of each candidate accelerator when running the target model through experiments or other means, and then set a fixed Inference_Latency for each candidate accelerator based on the collected inference latency.

[0068] S302: For each candidate accelerator: Determine the total inference latency of the candidate accelerator based on its Inference_Latency and the number of pending tasks, and normalize the total inference latency to obtain the inference latency parameter of the candidate accelerator.

[0069] For example, the total inference latency (Total_Latency) of the candidate accelerator can be determined using the formula Total_Latency = Inference_Latency + Queue_Waiting_Time.

[0070] Where Queue_Waiting_Time equals the product of Inference_Latency and the number of pending tasks already available on the candidate accelerator. Pending tasks on a candidate accelerator refer to tasks currently assigned to that accelerator and waiting to be processed.

[0071] Thus, since Queue_Waiting_Time equals the product of Inference_Latency and the number of pending tasks already processed by the candidate accelerator, i.e., Queue_Waiting_Time is the inference latency when running the target model on the candidate accelerator, and Inference_Latency is the inference latency of each existing pending task on the candidate accelerator, the estimated time required for the candidate accelerator to process each existing pending task can be obtained. Therefore, the total inference latency (Total_Latency) can be quickly estimated by performing simple calculations based on only two data points: Inference_Latency and the number of existing pending tasks on the candidate accelerator. This improves the efficiency of task scheduling decisions and, consequently, task scheduling efficiency.

[0072] Optionally, in the above formula Total_Latency = Inference_Latency + Queue_Waiting_Time, Queue_Waiting_Time can also be equal to the sum of the inference latency of the models to which each task belongs when the candidate accelerator runs.

[0073] For example, suppose that for candidate accelerator 1, there are currently tasks A, B, C, and D to be processed. Suppose that tasks A and B belong to model 1, task C belongs to model 2, and task D belongs to model 3. Suppose that the inference latency (Inference_Latency1) of candidate accelerator 1 when running model 1 is 1 second, the inference latency (Inference_Latency1) of candidate accelerator 1 when running model 2 is 2 seconds, and the inference latency (Inference_Latency1) of candidate accelerator 1 when running model 3 is 3 seconds. Then the Queue_Waiting_Time corresponding to candidate accelerator 1 is equal to 1+1+2+3=7 seconds.

[0074] At this point, a more accurate and reliable Queue_Waiting_Time can be obtained, which can improve the reliability of the identified target accelerator.

[0075] In one optional embodiment of this application, the total inference latency Total_Latency can be normalized according to the formula S_latency = (Latency_max - Total_Latency) / (Latency_max - Latency_min) to obtain the inference latency parameter S_latency of the candidate accelerator.

[0076] Where Latency_max is the maximum total inference delay among all candidate accelerators; and Latency_min is the minimum total inference delay among all candidate accelerators.

[0077] For example, suppose the candidate accelerators include candidate accelerator 1, candidate accelerator 2, and candidate accelerator 3, and suppose their respective Total_Latency is A, B, and C, respectively, where A > B > C. Then: the inference latency parameter S_latency of candidate accelerator 1 is (AA) / (AC) = 0; the inference latency parameter S_latency of candidate accelerator 2 is (AB) / (AC) (since AB is less than AC, (AB) / (AC) is a value greater than 0 and less than 1); the inference latency parameter S_latency of candidate accelerator 3 is (AC) / (A - C) = 1. It can be seen that, based on the above method, normalization can ensure that the obtained inference latency parameter S_latency is between 0 (inclusive) and 1 (inclusive), which is beneficial for more objectively combining it with energy consumption index values ​​to calculate the fit score, making the obtained fit score more objective. Furthermore, based on the inference latency parameter S_latency obtained from the above formula, the smaller the total inference latency of the candidate accelerator, the higher the inference latency parameter S_latency of the candidate accelerator. In this way, the inference latency parameter S_latency of each candidate accelerator can be positively correlated with the performance of each candidate accelerator in processing the target scheduled task, which is conducive to the determination of the adaptation score.

[0078] It is understood that the above is merely an example of one method for normalizing total inference latency in this application. Other normalization methods can also be used, such as using the formula (M - Total_Latency) / M, where M is a preset, relatively large constant. This application does not limit the normalization method for total inference latency.

[0079] In this embodiment, the performance index value can include, in addition to the inference latency parameter when the candidate accelerator runs the target model, other parameters that reflect the performance of the candidate accelerator running the target model can also be used to calculate it. For example, the processor utilization rate when the candidate accelerator runs the target model can be used. When calculating using the processor utilization rate when the candidate accelerator runs the target model, the processor utilization rate can be normalized to obtain a normalized parameter (denoted as the processor utilization rate parameter). The relevant normalization methods can be found above and will not be repeated here.

[0080] It is understandable that when a performance metric includes multiple normalized parameters, a weighted sum of these normalized parameters can be performed to obtain the final performance metric. For example, when both the inference latency parameter S_latency and the processor utilization parameter are included, a weighted sum of S_latency and processor utilization parameters can be performed to obtain the final performance metric.

[0081] In one optional embodiment of this application, the energy consumption index value for each of the candidate accelerators can be based on the following Figure 4 Calculated as shown: S401: Obtain the energy consumption when the candidate accelerator runs the target model.

[0082] In an optional embodiment of this application, the energy consumption (Energy) of any candidate accelerator running the target model can be an estimated value determined based on historical energy consumption records of the candidate accelerator running the target model in the past. For example, a heterogeneous system can continuously record the energy consumption of each candidate accelerator running the target model during operation; these energy consumptions are the historical energy consumptions. When S401 needs to be executed, these historical energy consumptions can be obtained, or the average value of historical energy consumption within the most recent preset time period or a preset number of times can be calculated to obtain the estimated energy consumption (Energy) of each candidate accelerator running the target model.

[0083] In another optional embodiment of this application, the energy consumption (Energy) of any candidate accelerator running the target model can be the pre-inference energy consumption recorded when using the candidate accelerator to perform pre-inference on the target model. For example, when the heterogeneous system has just started or the target model has just been loaded onto the candidate accelerator, the candidate accelerator can run the target model for one round of tasks, and the energy consumption at this time (i.e., pre-inference energy consumption) can be recorded. This recorded energy consumption can then be used as the energy consumption (Energy) of the candidate accelerator running the target model for subsequent calculations.

[0084] It is understood that the two optional implementation methods described above can be used simultaneously. When used simultaneously, the pre-inference energy consumption in the second optional implementation method can be used as the initial energy consumption, and the estimated value in the first optional implementation method can be used as the energy consumption after the target model is actually run on the candidate accelerator.

[0085] In another optional embodiment of this application, engineers may collect the energy consumption of each candidate accelerator when running the target model through experiments or other means, and then set a fixed energy consumption value for each candidate accelerator based on the collected energy consumption.

[0086] S402: Normalize the energy consumption Energy to obtain the energy consumption index value S_energy of the candidate accelerator.

[0087] For example, the energy consumption index value S_energy of the candidate accelerator can be determined according to the formula S_energy=(Energy_max - Energy) / (Energy_max –Energy_min).

[0088] Where Energy_max is the maximum energy consumption among the energy consumption of each candidate accelerator; and Energy_min is the minimum energy consumption among the energy consumption of each candidate accelerator.

[0089] For example, suppose the candidate accelerators include candidate accelerator 1, candidate accelerator 2, and candidate accelerator 3, and suppose their corresponding Energy values ​​are E1, E2, and E3, respectively, and E1 > E2 > E3. Then: the energy consumption index value of candidate accelerator 1 is S_energy = (E1 - E1) / (E1 - E3) = 0; the energy consumption index value of candidate accelerator 2 is S_energy = (E1 - E2) / (E1 - E3) (since E1 - E2 is less than E1 - E3, (E1 - E2) / (E1 - E3) is a value greater than 0 and less than 1); the energy consumption index value of candidate accelerator 3 is S_energy = (E1 - E3) / (E1 - E3) = 1. As can be seen, normalization based on the above method ensures that the obtained energy consumption index value S_energy is between 0 (inclusive) and 1 (inclusive), which facilitates a more objective calculation of the adaptation score by combining it with the performance index value, making the adaptation score more objective. Furthermore, based on the energy consumption index value S_energy obtained from the above formula, the lower the energy consumption of the candidate accelerator, the higher its energy consumption index value S_energy. Thus, the energy consumption index value S_energy of each candidate accelerator is positively correlated with its energy consumption performance when processing the target scheduled task, thereby facilitating the determination of the adaptation score.

[0090] It is understood that the above is merely an example of one method for normalizing energy consumption (Energy) in this application embodiment. Other normalization methods can also be used, such as normalization using the formula (E - Energy) / E, where E is a preset, relatively large constant. This application embodiment does not limit the method of normalizing energy consumption (Energy).

[0091] S202: Calculate the suitability score of each candidate accelerator for the target scheduled task based on the performance index and energy consumption index of each candidate accelerator.

[0092] In this embodiment, for each candidate accelerator, its performance index and energy consumption index can be weighted and summed to obtain a suitability score for each candidate accelerator for the target scheduled task. For example, assuming the performance index is the aforementioned inference latency parameter S_latency, the formula Score = α is used. S_latency +β S_energy (energy consumption index value) calculates the suitability score for each candidate accelerator for the target scheduled task. Here, α is the first weighting coefficient, corresponding to the performance index value, and β is the second weighting coefficient, corresponding to the energy consumption index value.

[0093] Optionally, the sum of α and β can be 1, and α and β can be numbers greater than or equal to 0.

[0094] In one optional embodiment of this application, a set of α and β can be pre-defined in the heterogeneous system for its use. When setting α and β, the settings can be differentiated according to the different computing devices on which the heterogeneous system resides. For example, in a scenario where the computing device is a server, α and β can be set to α > β; in a scenario where the computing device is a mobile terminal, α and β can be set to β > α.

[0095] In another optional embodiment of this application, multiple sets of α and β can be pre-defined in the heterogeneous system, and each set of α and β can be associated with different operating modes of the heterogeneous system, so as to realize the selection of α and β according to the current operating mode of the heterogeneous system.

[0096] For example, heterogeneous systems may have different operating modes during runtime, and different α and β values ​​can be set for different operating modes. For example, there is a performance-first mode (which tends to select accelerators with better performance, and when setting weighting coefficients, α > β), a high-performance mode (which only considers the accelerator with the best performance, and when setting weighting coefficients, α = 1 and β = 0), an energy-first mode (which tends to select accelerators with better energy efficiency, and when setting weighting coefficients, α < β), and an energy-saving mode (which only considers the accelerator with the best energy efficiency, and when setting weighting coefficients, α = 0 and β = 1).

[0097] When calculating the suitability score of each candidate accelerator for the target scheduling task, the current operating mode of the heterogeneous system can be obtained, and then the corresponding α and β values ​​can be acquired. Next, for each candidate accelerator, the performance and energy consumption values ​​are weighted and summed using α and β to obtain the suitability score of that candidate accelerator for the target scheduling task. In this way, for different operating modes of the heterogeneous system, a biased selection can be achieved, making the selected target accelerator more suitable for the current operating mode of the heterogeneous system.

[0098] Considering that in some heterogeneous systems, different processing modes can be implemented with fine granularity for different tasks, and different processing modes can correspond to different α and β, in another optional embodiment of this application, the processing mode of the heterogeneous system for the target task to be scheduled can be obtained, and then the α and β corresponding to the processing mode can be obtained. Then, for each candidate accelerator: the performance index value and the energy consumption index value are weighted and summed using α and β to obtain the adaptation score of the candidate accelerator for the target task to be scheduled.

[0099] For example, suppose the processing modes of a heterogeneous system can also include performance-first mode, high-performance mode, energy-first mode, and energy-saving mode. Suppose there are tasks A, B, C, and D, and task A needs to be processed using performance-first mode, task B using high-performance mode, task C using energy-first mode, and task D using energy-saving mode. Then, when the target task to be scheduled is task A, a set of α1 and β1 can be obtained, where α1 > β1. Based on α1 and β1, the suitability score of each candidate accelerator for task A can be calculated. When the target task to be scheduled is task B, a set of α2 and β2 can be obtained, where α2 equals 1 and β2 equals 0. Based on α2 and β2, the suitability score of each candidate accelerator for task B can be calculated. When the target task to be scheduled is task C, a set of α3 and β3 can be obtained, where α3 < β3. Based on α3 and β3, the suitability score of each candidate accelerator for task C can be calculated. When the target task to be scheduled is task D, a set of α4 and β4 can be obtained, where α4 equals 0 and β4 equals 1. Based on α4 and β4, the suitability score of each candidate accelerator for task D can then be calculated. This allows for different preference selections for different tasks, ensuring that the selected target accelerators better meet the processing needs of heterogeneous systems for the target task to be scheduled.

[0100] S203: Send the target task to be scheduled to the target accelerator with the highest adaptation score among the candidate accelerators.

[0101] In this embodiment, considering that many accelerators support batch processing, for example, iGPUs can deploy batch processing services with a batch size of 8, and GPGPUs can deploy batch processing services with a batch size of 32. Here, the value of `batch` represents the number of tasks that the accelerator needs to process simultaneously when performing batch processing. `batch=8` means that the accelerator can process 8 tasks simultaneously when performing batch processing, and `batch=32` means that the accelerator can process 32 tasks simultaneously when performing batch processing.

[0102] It's understandable that each accelerator can deploy services with a batch size of 1, in which case the accelerator supports processing tasks one by one. It's also understandable that when an accelerator processes tasks one by one, it can utilize more resources for a single task compared to batch processing, thus increasing the efficiency of a single round of task processing. However, when there are many tasks, batch processing can process multiple tasks simultaneously, so although the processing time per round is longer, the overall efficiency is higher.

[0103] Based on this, in one feasible implementation of the present application, all tasks can be scheduled one by one, and each accelerator can process each task one by one.

[0104] However, in one feasible implementation of this application, batch processing trigger conditions can be preset. Before obtaining the performance index values ​​and energy consumption index values ​​of each candidate accelerator for the target model to which the target scheduled task belongs, it is first determined whether the batch processing conditions have been triggered.

[0105] If the batch processing condition is not triggered, the execution accelerator processes tasks one by one. That is, at this time, the tasks currently to be scheduled in the preset total task queue can be identified as target tasks to be scheduled, and these target tasks to be scheduled can be assigned to the selected target accelerators.

[0106] The main task queue can be configured with the sequential scheduling order of its tasks, allowing tasks to be scheduled one by one according to the predefined order. The task to be scheduled currently is the task in the main task queue that needs to be scheduled when the tasks are scheduled according to the preset order.

[0107] When batch processing conditions are triggered, batch processing can be used. In this case, the N batch-processable tasks in the total task queue can be identified as target tasks to be scheduled, and these N tasks can be assigned together to the selected target accelerators.

[0108] The N tasks that can be batch-processed include the aforementioned tasks that currently need to be scheduled. The value of N can be equal to the batch size of the target accelerator. For example, if the selected target accelerator supports a batch size of 8, then the value of N can be 8; if the selected target accelerator supports a batch size of 32, then the value of N can be 32.

[0109] In this way, when the batch processing conditions are not triggered (such as when there are few tasks to be scheduled), confirming the target accelerator and scheduling tasks one by one can maximize the use of the target accelerator for task processing, thereby improving task processing efficiency. When there are many tasks to be scheduled, batch processing can be used to improve the overall task processing efficiency, thereby improving the overall resource utilization of the heterogeneous system.

[0110] Optionally, batch processing conditions may include at least one of the following: The minimum total inference latency (Latency_min) among the total inference latency of each accelerator when running the target model is higher than a preset first threshold. The minimum total inference latency Latency_min among the total inference latency when each accelerator runs the target model is higher than a preset first threshold, and the number of batch-processable tasks in the total task queue after a preset first time interval is greater than a preset second threshold. The number of batch-processable tasks in the total task queue is greater than the preset third threshold.

[0111] The first threshold, second threshold, and third threshold can be set according to actual needs, and the second threshold and third threshold are greater than or equal to the minimum value among the batch values ​​supported by each accelerator.

[0112] For example, suppose as Figure 1 In the heterogeneous system shown, the iGPUs are deployed with batch=1 and batch=8 services, the GPGPUs are deployed with batch=1 and batch=32 services, and the remaining accelerators are deployed with only batch=1. Therefore, the second and third thresholds should be values ​​greater than or equal to 8.

[0113] In this embodiment of the application, when the batch processing condition is triggered, the candidate accelerator should be: among all accelerators, the accelerator that supports a batch processing value less than or equal to the number of tasks that can be batch processed in the total task queue, and meets the preset accelerator participation scheduling conditions (such as no failure).

[0114] Taking the previous example again, assume that all accelerators meet the preset conditions for accelerator participation in scheduling.

[0115] For example, assuming the total number of batch-processable tasks in the task queue is 20, since 20 is greater than 8 and less than 32, the only accelerator that supports a batch size of 32 or less is the iGPU. Therefore, the iGPU is the candidate accelerator. If there is only one iGPU, it can be directly used as the target accelerator, and 8 batch-processable tasks can be assigned to it. Then, the batch processing condition determination logic mentioned above can be executed again. If there are multiple iGPUs, one iGPU can be identified as the target accelerator using the method described above. Then, 8 batch-processable tasks can be assigned to the target accelerator, and the batch processing condition determination logic mentioned above can be executed again.

[0116] For example, assuming the total number of batch-processable tasks in the task queue is 32, since the accelerators with a batch size less than or equal to 32 are iGPUs and GPGPUs, the candidate accelerators are iGPUs and GPGPUs. The method described earlier can then be used to determine one of the iGPUs and GPGPUs as the target accelerator. If the target accelerator is an iGPU, then 8 batch-processable tasks are assigned to that iGPU, and the aforementioned batch processing condition check logic is executed again. If the target accelerator is a GPGPU, then the 32 batch-processable tasks are assigned to that GPGPU, and the aforementioned batch processing condition check logic is executed again.

[0117] It is understood that in the aforementioned embodiment where the estimated value determined by the historical inference latency recorded when the candidate accelerator runs the target model is used as the inference latency (Inference_Latency) when the candidate accelerator runs the target model, it is possible to further distinguish between the historical inference latency of each candidate accelerator when running the target model with batch=1 and the historical inference latency of each candidate accelerator when running the target model with batch processing (batch not equal to 1).

[0118] Then, if the batch processing condition is not triggered, the estimated value determined by the historical inference latency of each candidate accelerator when running the target model with batch=1 is used as the inference latency (Inference_Latency) of the candidate accelerator when running the target model.

[0119] When batch processing conditions are triggered, the estimated value of the historical inference latency determined by each candidate accelerator when running the target model in batch (batch is not equal to 1) is used as the inference latency (Inference_Latency) when the candidate accelerator runs the target model.

[0120] It is understood that in the aforementioned embodiment where the pre-inference latency recorded when using the candidate accelerator to perform pre-inference on the target model is used as the inference latency (Inference_Latency) when the candidate accelerator runs the target model, it is possible to further distinguish between recording the pre-inference latency of each candidate accelerator when performing pre-inference on the target model with batch=1, and the pre-inference latency of each candidate accelerator when performing pre-inference on the target model with batch processing (batch not equal to 1).

[0121] Then, if the batch processing condition is not triggered, the pre-inference latency of each candidate accelerator performing pre-inference on the target model with batch=1 is used as the inference latency (Inference_Latency) when the candidate accelerator runs the target model.

[0122] When batch processing conditions are triggered, the pre-inference latency of each candidate accelerator performing pre-inference on the target model in batch processing (batch is not equal to 1) is used as the inference latency (Inference_Latency) when the candidate accelerator runs the target model.

[0123] In this way, more accurate performance index values ​​can be determined for both batch processing and non-batch processing scenarios.

[0124] It is understood that in the aforementioned embodiment where the estimated value determined based on the historical energy consumption recorded when the candidate accelerator runs the target model is used as the energy consumption when the candidate accelerator runs the target model, it is possible to further distinguish and record the historical energy consumption of each candidate accelerator when running the target model with batch=1, and the historical energy consumption of each candidate accelerator when running the target model with batch processing (batch not equal to 1).

[0125] Then, without triggering the batch processing condition, the estimated value determined by the historical energy consumption of each candidate accelerator when running the target model with batch=1 is used as the energy consumption of the candidate accelerator when running the target model.

[0126] When batch processing conditions are triggered, the estimated energy consumption of each candidate accelerator when running the target model in batch (batch not equal to 1) is used as the energy consumption of the candidate accelerator when running the target model.

[0127] It is understood that in the aforementioned embodiment where the pre-inference energy consumption recorded when using the candidate accelerator to perform pre-inference on the target model is used as the energy consumption when the candidate accelerator runs the target model, it is possible to further distinguish and record the pre-inference energy consumption of each candidate accelerator when performing pre-inference on the target model with batch=1, and the pre-inference energy consumption of each candidate accelerator when performing pre-inference on the target model with batch processing (batch not equal to 1).

[0128] Then, if the batch processing condition is not triggered, the energy consumption of each candidate accelerator for pre-inference of the target model in batch=1 is used as the energy consumption when the candidate accelerator runs the target model.

[0129] When batch processing conditions are triggered, the energy consumption of each candidate accelerator for pre-inference of the target model in batch processing (batch is not equal to 1) is used as the energy consumption when the candidate accelerator runs the target model.

[0130] In this way, more accurate energy consumption index values ​​can be determined for both batch processing and non-batch processing scenarios.

[0131] In one optional embodiment of this application, when batch processing conditions are triggered, the tasks that can be batch-processed in the total task queue are those belonging to the same target model. For example, assuming there are currently 200 tasks in the total task queue, of which 150 belong to model A and 50 belong to model B, and assuming that according to a preset scheduling order, the task to be scheduled is task 1, which belongs to model B, then the target model can be determined to be model B. Therefore, the 50 tasks belonging to model B can be determined to be batch-processable in the total task queue. In this way, after task allocation, when the target accelerator executes batch-processed tasks, it only needs to load one target model, which can better utilize the performance of the target accelerator.

[0132] In another optional embodiment of this application, if the accelerator can support parallel processing of multiple different models of tasks at the same time, then the batch-processable tasks in the total task queue can also be considered as all tasks in the total task queue.

[0133] In this embodiment of the application, if the batch processing condition is not triggered, all accelerators that meet the preset accelerator participation scheduling conditions can be selected as candidate accelerators.

[0134] It is understandable that in the actual operation of a heterogeneous system, one or more accelerators may become unsuitable for task processing due to factors such as failure or overload.

[0135] Therefore, in an optional embodiment of this application, task rescheduling conditions can be preset. After sending the target task to be scheduled to the target accelerator with the highest adaptation score among the candidate accelerators, it can be monitored whether the target accelerator has triggered the task rescheduling condition. If the task rescheduling condition is triggered, new candidate accelerators can be re-determined from among the accelerators (the new candidate accelerators no longer need to include the target accelerator). Then, the performance index and energy consumption index values ​​of each new candidate accelerator for the target model are re-acquired, and the adaptation score of each new candidate accelerator for the target task is calculated based on the performance index and energy consumption index values ​​of each new candidate accelerator. The target task to be scheduled is then re-sent to the accelerator with the highest adaptation score among the new candidate accelerators. In this way, when the original target accelerator is no longer suitable for executing the task, the task can be transferred to a new accelerator for execution, increasing the probability that the task can be executed correctly and improving the reliability of the heterogeneous system.

[0136] Optionally, the task rescheduling conditions may include at least one of the following: target accelerator failure; the target accelerator processing a single task for more than a preset second duration.

[0137] The preset second duration can be set by the engineer according to actual needs.

[0138] Based on the task scheduling method provided in this embodiment, by obtaining the performance index and energy consumption index values ​​of each candidate accelerator for the target model to which the target task belongs, and calculating the adaptation score of each candidate accelerator for the target task, the target task is sent to the target accelerator with the highest adaptation score for processing. In this way, when scheduling tasks, the adaptability of each candidate accelerator to the target task is comprehensively considered from both performance and energy consumption perspectives, making the target accelerator more suitable for the target task and enabling greater utilization of accelerator resources in the heterogeneous system, thus improving resource utilization.

[0139] It is understood that, in order to facilitate the description of the embodiments of this application, the following example of a visual model will be used as an example to illustrate the embodiments of this application.

[0140] See Figure 5 As shown, Figure 5 This paper illustrates a software architecture for implementing the solution of this application, including a task awareness layer, a scheduling decision layer, an inference engine layer, a feedback control layer, a resource monitoring layer, and underlying hardware for interfacing (e.g., GPGPU, NPU, iCPU, and other system resources such as memory). The CPU can execute the solution of this application based on this software architecture, specifically including: Step S1: Initialization and Model Registration For the task awareness layer of a heterogeneous system, there is a general task queue, and the tasks in the queue are scheduled according to subsequent scheduling strategies.

[0141] Each accelerator in the heterogeneous system loads the target visual model sequentially. Based on the computing power of each accelerator, different batches of inference services are deployed. For example, the NPU deploys inference services with batch=1, the iGPU deploys inference services with batch=1 and batch=8, and the GPGPU deploys inference services with batch=1 and batch=32.

[0142] When the heterogeneous system starts up or the target visual model is loaded, pre-inference is performed on the target visual model to obtain data such as its pre-inference latency and pre-inference energy consumption, and the target visual model is registered into the system.

[0143] Step S2: Real-time performance and energy consumption monitoring During the operation of the heterogeneous system, the resource monitoring layer continuously monitors the real-time status of each accelerator. This includes: recording the average inference latency (used as latency) for different batches of tasks being processed or recently processed by the target vision model; and obtaining the energy consumption of the accelerator to complete one task while executing the target vision model, using hardware sensors or estimation models. This data is then transmitted to the scheduling decision layer through the feedback control layer.

[0144] Step S3: Overall Score and Accelerator Selection When the scheduling decision layer receives a task to be scheduled, it queries the target visual model data collected in step S2 corresponding to the model identifier of the task. For each available accelerator (i.e., candidate accelerator), its comprehensive adaptation score Score = α is calculated. S_latency + β S_energy.

[0145] Here, S_latency is the inference latency parameter, representing the response speed; a higher value is better. The method for calculating S_latency is as follows: Estimate total inference latency: Total_Latency = Inference_Latency + Queue_Waiting_Time.

[0146] Where Inference_Latency is the inference latency of the accelerator in processing the target scheduled task itself; Queue_Waiting_Time is the estimated waiting time, which can be estimated as the product of Inference_Latency and the number of tasks that the accelerator already has to process.

[0147] Each accelerator calculates its Total_Latency, and then normalizes it to obtain S_latency for each accelerator = (Latency_max - Total_Latency) / (Latency_max - Latency_min).

[0148] S_energy is an energy consumption index value. The energy consumption of the current accelerator inference target to be scheduled is Energy. After normalizing the energy consumption Energy of all accelerators to process target to be scheduled, we have S_energy = (Energy_max - Energy) / (Energy_max – Energy_min).

[0149] α and β are configurable weighting coefficients, and α + β = 1. When α > β, performance is emphasized; when β > α, energy efficiency is emphasized. When α = 1 and β = 0, only inference latency is considered, and energy consumption is not considered.

[0150] Select the accelerator with the highest fit score as the target accelerator for this request.

[0151] Step S4: Dynamic Batch Processing Triggering and Selection If the minimum inference latency Latency_min calculated in S3 is less than or equal to the preset first threshold Group_begin_latency, then the scheduling strategy of S3 is followed, and the currently scheduled tasks in the total task queue are directly sent to the target accelerator for single-instance inference, and the process jumps to step S5.

[0152] When `Latency_min` is greater than `Group_begin_latency`, a short batch processing window is initiated (i.e., an interval of a preset first duration), for example, 10-50 milliseconds (this duration can be dynamically adjusted based on historical data and the accelerator). After the window ends, if the number of tasks in the total task queue exceeds the second threshold `Group_begin_length`, it is considered a high load, and batch processing is triggered. Similar to the method used in S3 for calculating the score, but this time the inference latency and energy consumption correspond to the monitored values ​​when batch = n (n>1). As in S3, for different batches supported by different accelerators, calculate their corresponding scores, and select the accelerator with the highest score and its corresponding batch to complete the inference task for this batch processing.

[0153] Step S5: Inference Execution Whether it's a single task or a batch task, the system will execute the task based on the finally selected target accelerator and its corresponding inference engine (model) in the inference engine layer. Specifically, the heterogeneous system will organize and adjust the input data format to match the requirements of the target accelerator, and execute the task through the pre-set inference engine on the target accelerator.

[0154] Based on the same inventive concept, this application also provides a task scheduling device 600 applied to a CPU in a heterogeneous system. Please refer to... Figure 6 As shown, Figure 6 It shows the use of Figure 2 The method illustrated is a task scheduling device. It should be understood that the specific functions of device 600 are described above; to avoid repetition, detailed descriptions are omitted here. Device 600 includes at least one software functional module that can be stored in memory or embedded in the operating system of device 600 in the form of software or firmware. Specifically: See Figure 6 As shown, the device 600 includes: an acquisition module 601, a calculation module 602, and a transmission module 603. Wherein: The acquisition module 601 is used to acquire the performance index value and energy consumption index value of each candidate accelerator for the target model to which the target scheduled task belongs; the candidate accelerators are at least two accelerators among all accelerators. The calculation module 602 is used to calculate the adaptation score of each candidate accelerator for the target scheduling task based on the performance index value and the energy consumption index value of each candidate accelerator. The sending module 603 is used to send the target task to be scheduled to the target accelerator with the highest adaptation score among the candidate accelerators.

[0155] In one feasible embodiment of this application, the performance index value includes the inference latency parameter when the candidate accelerator runs the target model; the acquisition module 601 is specifically used to: acquire the inference latency (Inference_Latency) when each candidate accelerator runs the target model; for each candidate accelerator: determine the total inference latency of the candidate accelerator based on the Inference_Latency of the candidate accelerator and the existing unprocessed tasks of the candidate accelerator; normalize the total inference latency to obtain the inference latency parameter of the candidate accelerator.

[0156] In one exemplary embodiment of this feasible implementation, the acquisition module 601 is specifically used to: determine the total inference latency Total_Latency of the candidate accelerator according to the formula Total_Latency = Inference_Latency + Queue_Waiting_Time; wherein: the Queue_Waiting_Time is equal to the product of the Inference_Latency and the number of pending tasks already present in the candidate accelerator.

[0157] In one exemplary embodiment of this feasible implementation, the acquisition module 601 is specifically used to: normalize the total inference latency Total_Latency according to the formula S_latency = (Latency_max - Total_Latency) / (Latency_max - Latency_min) to obtain the inference latency parameter S_latency of the candidate accelerators; wherein, Latency_max is the maximum total inference latency among the total inference latency of each candidate accelerator; and Latency_min is the minimum total inference latency among the total inference latency of each candidate accelerator.

[0158] In one exemplary embodiment of this feasible implementation, the inference latency (Inference_Latency) of each candidate accelerator when running the target model is: an estimated value determined based on the historical inference latency recorded when the candidate accelerator has historically run the target model; or, the pre-inference latency recorded when the candidate accelerator performs pre-inference on the target model.

[0159] In one feasible embodiment of this application, the acquisition module 601 is specifically used for: for each candidate accelerator: acquiring the energy consumption of the candidate accelerator when running the target model; normalizing the energy consumption to obtain the energy consumption index value of the candidate accelerator.

[0160] In one exemplary embodiment of this feasible implementation, the energy consumption of the candidate accelerator when running the target model is: an estimated value determined based on the historical energy consumption recorded when the candidate accelerator has historically run the target model; or, the pre-inference energy consumption recorded when the candidate accelerator is used to perform pre-inference on the target model.

[0161] In one exemplary embodiment of this feasible implementation, the acquisition module 601 is specifically used to: normalize the energy consumption Energy according to the formula S_energy=(Energy_max - Energy) / (Energy_max –Energy_min) to obtain the energy consumption index value S_energy of the candidate accelerator; wherein, Energy_max is the maximum energy consumption among the energy consumption of each candidate accelerator; and Energy_min is the minimum energy consumption among the energy consumption of each candidate accelerator.

[0162] In one feasible embodiment of this application, the calculation module 602 is specifically used to: obtain the current operating mode of the heterogeneous system; obtain the first weight coefficient and the second weight coefficient corresponding to the current operating mode; and for each candidate accelerator: use the first weight coefficient and the second weight coefficient to perform a weighted summation of the performance index value and the energy consumption index value to obtain the adaptation score of the candidate accelerator for the target scheduled task.

[0163] In another feasible embodiment of this application, the calculation module 602 is specifically used to: obtain the processing mode of the heterogeneous system for the target task to be scheduled; obtain the first weight coefficient and the second weight coefficient corresponding to the processing mode; for each candidate accelerator: use the first weight coefficient and the second weight coefficient to perform a weighted summation of the performance index value and the energy consumption index value to obtain the adaptation score of the candidate accelerator for the target task to be scheduled.

[0164] In one feasible embodiment of this application, the device 600 may further include: a batch processing decision module, configured to determine whether a preset batch processing condition is triggered before the acquisition module 601 acquires the performance index value and energy consumption index value of each candidate accelerator for the target model to which the target scheduled task belongs; if the batch processing condition is not triggered, then the currently scheduled task in the preset total task queue is determined as the target scheduled task; the currently scheduled task is: the task currently scheduled in the total task queue when scheduled according to a preset sequential scheduling order; the target model is the model to which the currently scheduled task belongs; if the batch processing condition is triggered, then N batch-processable tasks in the total task queue are determined as the target scheduled task; the target scheduled task includes the currently scheduled task.

[0165] In this feasible embodiment, the batch processing conditions include at least one of the following: the minimum total inference latency (Latency_min) among the total inference latency of each accelerator running the target model is higher than a preset first threshold; the minimum total inference latency (Latency_min) among the total inference latency of each accelerator running the target model is higher than the preset first threshold, and the number of batchable tasks in the total task queue after a preset first time interval is greater than a preset second threshold; the number of batchable tasks in the total task queue is greater than a preset third threshold; and the second threshold and the third threshold are greater than or equal to the minimum value among the batch values ​​supported by each accelerator.

[0166] In this feasible implementation, when the batch processing condition is triggered, the candidate accelerator is: among all accelerators, the accelerator that supports a batch value less than or equal to the number of tasks that can be batch processed in the total task queue, and meets the preset accelerator participation scheduling conditions.

[0167] In this feasible implementation, the tasks that can be batch-processed in the total task queue are those that belong to the target model.

[0168] In this feasible embodiment, if the batch processing condition is not triggered, the acquisition module 601 is further configured to determine all accelerators that meet the preset accelerator participation scheduling conditions as the candidate accelerators before acquiring the performance index value and energy consumption index value of each candidate accelerator for the target model to which the target scheduled task belongs.

[0169] In one feasible embodiment of this application, the device 600 further includes a reselection module, which is used to re-determine a new candidate accelerator from each of the accelerators in response to the triggering of a preset task rescheduling condition after the sending module 603 sends the target task to be scheduled to the target accelerator with the highest adaptation score among the candidate accelerators; wherein the new candidate accelerators do not include the target accelerator. The acquisition module 601 is also used to acquire the performance index values ​​and energy consumption index values ​​of each of the new candidate accelerators for the target model; The calculation module 602 is further configured to calculate the adaptation score of each of the new candidate accelerators for the target scheduling task based on the performance index value and the energy consumption index value of each of the new candidate accelerators. The sending module 603 is also used to resend the target task to be scheduled to the accelerator with the highest adaptation score among the new candidate accelerators.

[0170] In this feasible embodiment, the task rescheduling condition includes at least one of the following: the target accelerator malfunctions; the target accelerator processes a single task for a duration exceeding a preset second duration.

[0171] It should be understood that, for the sake of brevity, some of the content described in the method embodiment apparatus will not be repeated in the apparatus embodiment.

[0172] This application provides a computing device comprising a heterogeneous system and a memory, wherein the heterogeneous system includes a CPU and at least two accelerators connected to the CPU. Wherein: The CPU is used to execute one or more programs stored in memory to implement the task scheduling method described above.

[0173] The memory can be RAM (Random Access Memory), ROM (Read-Only Memory), or flash memory. The above are merely examples and should not be construed as limiting this application.

[0174] It can also be understood that computing devices may include more components. For example, they may have an internal communication bus for communication between the CPU and memory; they may also have external communication interfaces, such as USB (Universal Serial Bus) interfaces, CAN (Controller Area Network) bus interfaces, etc.; and they may also have information display components such as displays, but this is not a limitation.

[0175] This application also provides a computer-readable storage medium, such as a floppy disk, optical disk, hard disk, flash memory, USB flash drive, SD (Secure Digital Memory Card), MMC (Multimedia Card), etc., in which one or more programs implementing the above steps are stored. These one or more programs can be executed by the CPU to implement the above task scheduling method. Further details will not be elaborated here.

[0176] This application also provides a computer program product, which includes a computer program that is executed by the CPU to implement the above-described task scheduling method.

[0177] In the embodiments of this application, the computer program product may be, but is not limited to, firmware installation packages, software installation packages, etc.

[0178] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the interconnections shown or discussed may be through interfaces, and may be electrical, mechanical, or other forms.

[0180] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0181] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0182] In this article, "multiple" refers to two or more.

[0183] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A task scheduling method, characterized in that, The method is applied to a central processing unit (CPU) in a heterogeneous system, wherein the heterogeneous system further includes at least two accelerators connected to the CPU; the method includes: Obtain the performance index and energy consumption index of each candidate accelerator for the target model of the target task to be scheduled; the candidate accelerators are at least two accelerators from all accelerators; The suitability score of each candidate accelerator for the target scheduling task is calculated based on the performance index value and energy consumption index value of each candidate accelerator. The target task to be scheduled is sent to the target accelerator with the highest adaptation score among the candidate accelerators.

2. The task scheduling method as described in claim 1, characterized in that, The performance metric values ​​include the inference latency parameters when the candidate accelerator runs the target model; Obtain the performance metrics of each candidate accelerator for the target model of the target task to be scheduled, including: Obtain the inference latency (Inference_Latency) when each of the candidate accelerators runs the target model; For each candidate accelerator: the total inference latency of the candidate accelerator is determined based on its Inference_Latency and the existing unprocessed tasks; the total inference latency is normalized to obtain the inference latency parameter of the candidate accelerator.

3. The task scheduling method as described in claim 2, characterized in that, Based on the Inference_Latency of the candidate accelerator and the existing unprocessed tasks of the candidate accelerator, the total inference latency of the candidate accelerator is determined, including: The total inference latency (Total_Latency) of the candidate accelerator is determined according to the formula Total_Latency = Inference_Latency + Queue_Waiting_Time. Wherein: Queue_Waiting_Time is equal to the product of Inference_Latency and the number of pending tasks already available for the candidate accelerator.

4. The task scheduling method as described in claim 2, characterized in that, The total inference delay is normalized to obtain the inference delay parameters of the candidate accelerators, including: The total inference latency Total_Latency is normalized according to the formula S_latency = (Latency_max - Total_Latency) / (Latency_max - Latency_min) to obtain the inference latency parameter S_latency of the candidate accelerator; Wherein, Latency_max is the maximum total inference delay among the total inference delays of each candidate accelerator; and Latency_min is the minimum total inference delay among the total inference delays of each candidate accelerator.

5. The task scheduling method as described in claim 2, characterized in that, The inference latency (Inference_Latency) when each of the candidate accelerators runs the target model is: The estimated value is determined based on the historical inference delay recorded when the target model was run by the candidate accelerator in the past; Alternatively, the pre-inference latency recorded when using the candidate accelerator to perform pre-inference on the target model.

6. The task scheduling method as described in claim 1, characterized in that, Obtain the energy consumption metrics of each candidate accelerator for the target model of the target task to be scheduled, including: For each of the candidate accelerators: Obtain the energy consumption of the candidate accelerator when running the target model; The energy consumption is normalized to obtain the energy consumption index value of the candidate accelerator.

7. The task scheduling method as described in claim 6, characterized in that, The energy consumption of the candidate accelerator when running the target model is: The estimated value is determined based on the historical energy consumption recorded when the target model was run by the candidate accelerator. Alternatively, the pre-inference energy consumption recorded when using the candidate accelerator to perform pre-inference on the target model.

8. The task scheduling method as described in claim 6, characterized in that, The energy consumption is normalized to obtain the energy consumption index value of the candidate accelerator, including: The energy consumption Energy is normalized according to the formula S_energy=(Energy_max - Energy) / (Energy_max – Energy_min) to obtain the energy consumption index value S_energy of the candidate accelerator; Wherein, Energy_max is the maximum energy consumption among the energy consumption of each candidate accelerator; and Energy_min is the minimum energy consumption among the energy consumption of each candidate accelerator.

9. The task scheduling method as described in claim 1, characterized in that, Based on the performance index and energy consumption index of each candidate accelerator, a suitability score for each candidate accelerator for the target scheduled task is calculated, including: Obtain the current operating mode of the heterogeneous system; Obtain the first weight coefficient and the second weight coefficient corresponding to the current operating mode; For each candidate accelerator: the performance index value and the energy consumption index value are weighted and summed using the first weighting coefficient and the second weighting coefficient to obtain the suitability score of the candidate accelerator for the target scheduled task.

10. The task scheduling method as described in claim 1, characterized in that, Based on the performance index and energy consumption index of each candidate accelerator, a suitability score for each candidate accelerator for the target scheduled task is calculated, including: Obtain the processing mode of the heterogeneous system for the target task to be scheduled; Obtain the first weight coefficient and the second weight coefficient corresponding to the processing mode; For each candidate accelerator: the performance index value and the energy consumption index value are weighted and summed using the first weighting coefficient and the second weighting coefficient to obtain the suitability score of the candidate accelerator for the target scheduled task.

11. The task scheduling method according to any one of claims 1-10, characterized in that, Before obtaining the performance and energy consumption metrics of each candidate accelerator for the target model of the target task to be scheduled, the method further includes: Determine whether the preset batch processing conditions have been triggered; If the batch processing condition is not triggered, the task currently to be scheduled in the preset total task queue is determined as the target task to be scheduled; the task currently to be scheduled is: the task currently to be scheduled in the total task queue when scheduled in a preset sequential order; the target model is the model to which the task currently to be scheduled belongs. If the batch processing condition is triggered, the N batch-processable tasks in the total task queue are determined as the target tasks to be scheduled; the target tasks to be scheduled include the currently scheduled task.

12. The task scheduling method as described in claim 11, characterized in that, The batch processing conditions include at least one of the following: The minimum total inference latency (Latency_min) among the total inference latency of each accelerator running the target model is higher than a preset first threshold. The minimum total inference latency Latency_min among the total inference latency of each accelerator when running the target model is higher than a preset first threshold, and the number of batch-processable tasks in the total task queue after a preset first time interval is greater than a preset second threshold. The number of batch-processable tasks in the total task queue is greater than a preset third threshold. The second threshold and the third threshold are greater than or equal to the minimum of the batch values ​​supported by each of the accelerators.

13. The task scheduling method as described in claim 11, characterized in that, When the batch processing condition is triggered, the candidate accelerator is: among all accelerators, the accelerator that supports a batch value less than or equal to the number of tasks that can be batch processed in the total task queue, and meets the preset accelerator participation scheduling conditions.

14. The task scheduling method as described in claim 13, characterized in that, The tasks that can be batch-processed in the overall task queue are those that belong to the target model.

15. The task scheduling method as described in claim 11, characterized in that, If the batch processing condition is not triggered, before obtaining the performance index and energy consumption index values ​​of each candidate accelerator for the target model to which the target scheduled task belongs, the method further includes: Among all the accelerators, those that meet the preset accelerator participation scheduling conditions are identified as the candidate accelerators.

16. The task scheduling method according to any one of claims 1-10, characterized in that, After sending the target task to be scheduled to the target accelerator with the highest adaptation score among the candidate accelerators, the method further includes: In response to the triggering of a preset task rescheduling condition, a new candidate accelerator is re-determined from each of the accelerators; wherein the target accelerator is not included in the new candidate accelerators. Obtain the performance and energy consumption values ​​of each of the new candidate accelerators for the target model; Based on the performance index value and energy consumption index value of each of the new candidate accelerators, the adaptation score of each of the new candidate accelerators for the target scheduling task is calculated respectively. The target task to be scheduled is then resent to the accelerator with the highest adaptation score among the new candidate accelerators.

17. The task scheduling method as described in claim 16, characterized in that, The task rescheduling conditions include at least one of the following: The target accelerator malfunctioned; The target accelerator takes longer than a preset second duration to process a single task.

18. A task scheduling device, characterized in that, The task scheduling device is applied to a central processing unit (CPU) in a heterogeneous system, wherein the heterogeneous system further includes at least two types of accelerators connected to the CPU; the task scheduling device includes: The acquisition module is used to acquire the performance index and energy consumption index of each candidate accelerator for the target model to which the target scheduled task belongs; the candidate accelerators are at least two accelerators among all accelerators. The calculation module is used to calculate the adaptation score of each candidate accelerator for the target scheduling task based on the performance index value and the energy consumption index value of each candidate accelerator. The sending module is used to send the target task to be scheduled to the target accelerator with the highest adaptation score among the candidate accelerators.

19. A computing device, characterized in that, include: A heterogeneous system and memory, the heterogeneous system including a central processing unit (CPU) and at least two accelerators connected to the CPU; the CPU is used to execute one or more programs stored in the memory to implement the task scheduling method as described in any one of claims 1-17.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by a central processing unit (CPU) to implement the task scheduling method as described in any one of claims 1-17.

21. A computer program product, characterized in that, The computer program product includes a computer program that, when run by a central processing unit (CPU), causes the CPU to execute the task scheduling method according to any one of claims 1-17.

Images