Scheduling method and device, equipment and storage medium

Abstract

The invention relates to a scheduling method and device, equipment and a storage medium. Aiming at the model reasoning task, screening out a plurality of first computing power units from the computing power unit set; splitting a model required to be used for executing the model reasoning task into a plurality of sub-models; at least aiming at enabling the task completion time of the plurality of first computing power units to be consistent or basically consistent, allocating the plurality of sub-models to the plurality of first computing power units, so that the first computing power units execute computing tasks corresponding to the sub-models allocated to the first computing power units; and obtaining a reasoning result of the model reasoning task based on calculation results of the plurality of first computing power units. Therefore, the first computing power unit distributed with the sub-model can synchronously or basically synchronously complete the task, so that the overall waiting time caused by the delay of a single computing power unit is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a scheduling method, apparatus, device and storage medium. Background Art

[0002] With the development of technologies such as artificial intelligence (AI) and large language models, the demand for computing resources for model reasoning tasks is increasing, which requires the use of multiple computing devices to complete model reasoning tasks.

[0003] However, when faced with specific model reasoning tasks, how to select appropriate computing resources from available computing resources to complete the model reasoning task is a problem that needs to be solved urgently. Summary of the invention

[0004] A technical problem to be solved by the present disclosure is how to select appropriate computing resources from available computing resources to complete the model reasoning task.

[0005] According to the first aspect of the present disclosure, a scheduling method is provided, including: for a model reasoning task, screening out a plurality of first computing power units from a set of computing power units; splitting a model required to execute the model reasoning task into a plurality of sub-models; at least with the goal of making the task completion times of the plurality of first computing power units consistent or substantially consistent, allocating the plurality of sub-models to the plurality of first computing power units, so that the first computing power units execute computing tasks corresponding to the sub-models allocated thereto; and obtaining an inference result of the model reasoning task based on the computing results of the plurality of first computing power units.

[0006] Optionally, at least with the goal of making the task completion times of the multiple first computing units consistent or substantially consistent, the multiple sub-models are assigned to the multiple first computing units, including: at least with the goal of minimizing the overall task completion time difference, the multiple sub-models are assigned to the multiple first computing units, and the overall task completion time difference is used to characterize the difference between the first task completion time of the first operator unit that completes the calculation first and the second task completion time of the first operator unit that completes the calculation last.

[0007] Optionally, the method also includes: predicting the calculation time of each of the first computing power units for each of the sub-models, the calculation time being used to characterize the time required for the first computing power unit to perform the computing task corresponding to the sub-model; at least with the goal of making the task completion time of the multiple first computing power units consistent or basically consistent, allocating the multiple sub-models to the multiple first computing power units, including: allocating the multiple sub-models to the multiple first computing power units based on the calculation time of each of the first computing power units for each of the sub-models, wherein the total calculation time of each of the first computing power units to which the sub-models are allocated is consistent or basically consistent, and the total calculation time is used to characterize the sum of the calculation times of the first computing power unit for all sub-models allocated to it.

[0008] Optionally, for a model reasoning task, multiple first computing power units are selected from a computing power unit set, including: for at least some of the computing power units in the computing power unit set, calculating the allocation score of each of the computing power units relative to the model reasoning task; and selecting multiple computing power units with top allocation scores from at least some of the computing power units to obtain the multiple first computing power units.

[0009] Optionally, the allocation score is positively correlated with one or more of a basic score, a task fitness, a load balancing factor, and a priority weight, wherein the basic score is used to characterize the actual performance of the computing power unit, the task fitness is used to characterize the degree of adaptation between the computing power unit and the model reasoning task, the load balancing factor is used to characterize the degree of load balancing of the computing power unit, and the priority weight is used to characterize the weight of the priority of the computing power unit and / or the model reasoning task.

[0010] Optionally, the basic score is positively correlated with the current availability score and the current computing power score, respectively. The method also includes: using a pre-built machine learning model to predict the current availability score and the current computing power score of each computing power unit, the input of the machine learning model is the status information of the computing power unit, and the output of the machine learning model is the current availability score and the current computing power score.

[0011] Optionally, the status information includes at least one of the following: type of computing unit; floating-point computing capability; memory usage; average utilization; host bandwidth; average network I / O; average task completion time; average energy consumption; error rate; average operating temperature; current load status; and elapsed running time.

[0012] Optionally, the method also includes: training multiple machine learning models based on historical data sets; evaluating the model effects of the multiple machine learning models; wherein, using pre-built machine learning models, predicting the current availability score and current computing power capacity score of each computing power unit, including: using the machine learning model with the best model effect performance to predict the current availability score and current computing power capacity score of each computing power unit.

[0013] Optionally, the task suitability is positively correlated with the historical performance of the computing unit under the same computing type as the model reasoning task, and negatively correlated with the best historical performance of the computing unit under all computing types, and the historical performance includes one or more of average task completion speed, success rate, and energy efficiency.

[0014] Optionally, the method further includes: updating the historical performance of the first computing unit after the first computing unit completes the calculation.

[0015] Optionally, the load balancing factor is negatively correlated with the current load of the computing power unit, and positively correlated with the maximum load of the computing power unit.

[0016] Optionally, the priority weight is positively correlated with the priority of the model reasoning task, and negatively correlated with the maximum priority of all currently pending model reasoning tasks; and / or the priority weight is positively correlated with the priority of the computing power unit.

[0017] Optionally, the method also includes: based on the memory requirements and / or computing type of the model reasoning task, selecting computing power units that meet the memory requirements and / or match the computing type from the set of computing power units to obtain at least part of the computing power units.

[0018] Optionally, the method further includes: monitoring performance changes of the computing power unit; and executing a dynamic adjustment strategy for the computing power unit that matches the performance changes of the computing power unit.

[0019] Optionally, the dynamic adjustment strategy includes at least one of the following: adjusting the priority of the computing power unit in the allocation process; adjusting the complex task allocation ratio of the computing power unit; adjusting the energy management strategy of the computing power unit; adjusting the cooling resource allocation strategy of the computing power unit; and adjusting the maintenance and inspection strategy of the computing power unit.

[0020] Optionally, the method also includes: after allocating the sub-model to the first computing power unit, updating the status information of the first computing power unit, and / or re-evaluating the performance of the first computing power unit using a machine learning model.

[0021] Optionally, the method also includes: determining whether the difference in computing progress between different first computing power units is greater than a first threshold; if the difference in computing progress between different first computing power units is greater than or equal to the first threshold, analyzing whether the performance benefit of allocating the unexecuted sub-model in the first computing power unit with relatively lagging computing progress to the first computing power unit with relatively leading computing progress is greater than the migration cost; if the performance benefit is greater than the migration cost, allocating the unexecuted sub-model in the first computing power unit with relatively lagging computing progress to the first computing power unit with relatively leading computing progress.

[0022] Optionally, the computing power unit set includes multiple heterogeneous computing power units.

[0023] Optionally, the model reasoning task is a text generation task; based on the calculation results of the multiple first computing power units, the reasoning result of the model reasoning task is obtained, including: based on the calculation results of the multiple first computing power units in each round of reasoning process, the words generated in the current round are obtained; the words obtained from multiple rounds of reasoning processes are combined in chronological order to obtain the text generation result corresponding to the text generation task.

[0024] According to a second aspect of the present disclosure, a scheduling device is provided, comprising: a screening module, for screening out a plurality of first computing power units from a set of computing power units for a model reasoning task; a splitting module, for splitting a model required for executing the model reasoning task into a plurality of sub-models; an allocation module, for allocating the plurality of sub-models to the plurality of first computing power units, at least with the goal of making the task completion times of the plurality of first computing power units consistent or substantially consistent, so that the first computing power units execute computing tasks corresponding to the sub-models allocated thereto; and a processing module, for obtaining an inference result of the model reasoning task based on the computing results of the plurality of first computing power units.

[0025] Optionally, the allocation module allocates the multiple sub-models to the multiple first computing units with the goal of at least minimizing the overall task completion time difference, and the overall task completion time difference is used to characterize the difference between the first task completion time of the first operator unit that completes the calculation first and the second task completion time of the first operator unit that completes the calculation last.

[0026] Optionally, the device also includes: a first prediction module, used to predict the calculation time of each of the first computing power units for each of the sub-models, and the calculation time is used to characterize the time required for the first computing power unit to perform the computing task corresponding to the sub-model; the allocation module allocates the multiple sub-models to the multiple first computing power units based on the calculation time of each of the first computing power units for each of the sub-models, wherein the total calculation time of each first computing power unit to which a sub-model is allocated is consistent or basically consistent, and the total calculation time is used to characterize the sum of the calculation times of the first computing power unit for all sub-models allocated to it.

[0027] Optionally, the screening module includes: a calculation module, used to calculate the allocation score of each computing power unit relative to the model reasoning task for at least part of the computing power units in the computing power unit set; and a screening sub-module, used to screen out multiple computing power units with high allocation scores from at least part of the computing power units to obtain the multiple first computing power units.

[0028] Optionally, the allocation score is positively correlated with one or more of a basic score, a task fitness, a load balancing factor, and a priority weight, wherein the basic score is used to characterize the actual performance of the computing power unit, the task fitness is used to characterize the degree of adaptation between the computing power unit and the model reasoning task, the load balancing factor is used to characterize the degree of load balancing of the computing power unit, and the priority weight is used to characterize the weight of the priority of the computing power unit and / or the model reasoning task.

[0029] Optionally, the basic score is positively correlated with the current availability score and the current computing power score, respectively, and the device also includes: a second prediction module, used to use a pre-built machine learning model to predict the current availability score and the current computing power score of each computing power unit, the input of the machine learning model is the status information of the computing power unit, and the output of the machine learning model is the current availability score and the current computing power score.

[0030] Optionally, the status information includes at least one of the following: type of computing unit; floating-point computing capability; memory usage; average utilization; host bandwidth; average network I / O; average task completion time; average energy consumption; error rate; average operating temperature; current load status; and elapsed running time.

[0031] Optionally, the device also includes: a training module for training multiple machine learning models based on historical data sets; an evaluation module for evaluating the model effects of the multiple machine learning models; and the second prediction module uses the machine learning model with the best model effect performance to predict the current availability score and current computing power capacity score of each computing power unit.

[0032] Optionally, the task suitability is positively correlated with the historical performance of the computing unit under the same computing type as the model reasoning task, and negatively correlated with the best historical performance of the computing unit under all computing types, and the historical performance includes one or more of average task completion speed, success rate, and energy efficiency.

[0033] Optionally, the device also includes: a first updating module, used to update the historical performance of the first computing unit after the first computing unit completes the calculation.

[0034] Optionally, the load balancing factor is negatively correlated with the current load of the computing power unit, and positively correlated with the maximum load of the computing power unit.

[0035] Optionally, the priority weight is positively correlated with the priority of the model reasoning task, and negatively correlated with the maximum priority of all currently pending model reasoning tasks; and / or the priority weight is positively correlated with the priority of the computing power unit.

[0036] Optionally, the device also includes: a selection module, which is used to select computing power units that meet the memory requirements and / or match the computing type from the computing power unit set based on the memory requirements and / or computing type of the model reasoning task, to obtain at least part of the computing power units.

[0037] Optionally, the device further includes: a monitoring module for monitoring performance changes of the computing power unit; and an adjustment module for executing a dynamic adjustment strategy matching the performance changes of the computing power unit.

[0038] Optionally, the dynamic adjustment strategy includes at least one of the following: adjusting the priority of the computing power unit in the allocation process; adjusting the complex task allocation ratio of the computing power unit; adjusting the energy management strategy of the computing power unit; adjusting the cooling resource allocation strategy of the computing power unit; and adjusting the maintenance and inspection strategy of the computing power unit.

[0039] Optionally, the device also includes: a second update module, used to update the status information of the first computing power unit after the sub-model is assigned to the first computing power unit, and / or re-evaluate the performance of the first computing power unit using a machine learning model.

[0040] Optionally, the device also includes: a judgment module, used to judge whether the difference between the computing progress of different first computing power units is greater than a first threshold; an analysis module, used to analyze whether the performance benefit of allocating the unexecuted sub-model in the first computing power unit with relatively lagging computing progress to the first computing power unit with relatively leading computing progress is greater than the migration cost if the difference between the computing progress of different first computing power units is greater than or equal to the first threshold; a migration module, used to allocate the unexecuted sub-model in the first computing power unit with relatively lagging computing progress to the first computing power unit with relatively leading computing progress if the performance benefit is greater than the migration cost.

[0041] Optionally, the computing power unit set includes multiple heterogeneous computing power units.

[0042] Optionally, the model reasoning task is a text generation task; the processing module obtains the words generated in the current round based on the calculation results of the multiple first computing units in each round of reasoning process, and combines the words obtained in multiple rounds of reasoning processes in chronological order to obtain a text generation result corresponding to the text generation task.

[0043] According to a third aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable codes are stored, and when the executable codes are executed by the processor, the processor executes the method described in the first aspect above.

[0044] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising an executable code, which, when executed by a processor of an electronic device, causes the processor to execute the method described in the first aspect above.

[0045] According to a fifth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect above.

[0046] The present disclosure selects multiple first computing units from a computing unit set for a model reasoning task, splits the model required to perform the model reasoning task into multiple sub-models, and assigns the multiple sub-models to the multiple first computing units with the goal of at least making the task completion time of the multiple first computing units consistent or substantially consistent. In this way, the first computing units assigned with the sub-models can complete the tasks synchronously or substantially synchronously, thereby reducing the overall waiting time caused by the delay of a single computing unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0048] Figure 1 is a schematic flow chart showing a scheduling method according to an embodiment of the present disclosure.

[0049] Figure 2 is an example diagram showing historical performance data according to one embodiment of the present disclosure.

[0050] Figure 3 It is a schematic diagram showing the overall architecture of a scheduling system according to a disclosed embodiment.

[0051] Figure 4 It is a schematic diagram showing the structure of a scheduling device according to an embodiment of the present disclosure.

[0052] Figure 5 A schematic diagram of the structure of a computing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0053] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0054] Tasks completed with the help of the reasoning capabilities of models (i.e. AI models) can be collectively referred to as model reasoning tasks.

[0055] Model reasoning tasks cover a wide range of tasks, including but not limited to classification tasks (such as image classification tasks, text classification tasks), computer vision target detection tasks, semantic segmentation tasks, natural language processing tasks (such as text generation tasks, named entity recognition tasks, question-answering tasks), multimodal tasks, etc.

[0056] The present disclosure is applicable to any model reasoning task that has a high demand for computing resources. The level of computing resource demand of the model reasoning task is related to the model used by the model reasoning task. Generally speaking, the number of parameters of the model directly determines the size and amount of calculation of the model. The more parameters a model has, the more parameters need to be loaded and calculated during reasoning, thus requiring more computing resources. Therefore, the present disclosure can be applicable to model reasoning tasks that use a large number of parameters of the model. For example, a text generation task based on a large language model (LLM).

[0057] Figure 1 is a schematic flow chart showing a scheduling method according to an embodiment of the present disclosure. Figure 1 The method shown can be implemented in software via a computer program, and can also be executed via a specifically configured computing device.

[0058] This disclosure is for Figure 1 There is no limitation on the order of execution of step S110 and step S120. That is, step S110 may be executed first, and then step S120; step S120 may be executed first, and then step S110; step S110 and step S120 may also be executed simultaneously without any particular order.

[0059] See also Figure 1 , in step S110, for the model inference task, multiple first computing units are selected from the computing unit set.

[0060] When there are multiple model reasoning tasks to be processed at the same time, corresponding priorities can be assigned to these multiple model reasoning tasks, and tasks can be scheduled based on the priorities. For example, model reasoning tasks can be selected one by one in order of priority from high to low for scheduling. For another example, multiple model reasoning tasks can also be scheduled at the same time, and the priority of the model reasoning tasks can be considered during the scheduling process, so that model reasoning tasks with relatively high priorities can be preferentially allocated to excellent computing resources.

[0061] The computing power unit set is used to characterize the set of currently available computing power units. For example, it can be a set of registered computing power units. The present disclosure supports dynamic configuration of computing power unit sets. For example, new computing power units can be added (i.e., registered) to the computing power unit set, and computing power units with problems in the computing power unit set can also be deleted (i.e., offline). A computing power unit refers to a device that can provide computing power resources. A computing power unit may also be called a computing unit or a computing device.

[0062] For computing units newly added to the computing unit set, their performance can be automatically evaluated and taken into consideration for scheduling.

[0063] Different types of model reasoning tasks have different computing resource demand characteristics. For example, some model reasoning tasks are CPU-intensive tasks, and some model reasoning tasks are GPU-intensive tasks. In order to be able to flexibly and efficiently handle more different types of model reasoning tasks, the computing power unit set can be a heterogeneous computing environment. That is, the computing power unit set can include multiple heterogeneous computing power units. For example, the computing power unit set can include a variety of heterogeneous computing devices including different types of CPUs (such as x86, ARM) and GPUs (such as NVIDIA, Huawei Ascend, Google TPU, etc.). As a result, the present disclosure can adapt to computing devices of different types and architectures, thereby providing enterprises with greater flexibility in hardware selection.

[0064] The first computing unit selected from the computing unit set is a computing unit suitable for processing model reasoning tasks. The number of first computing units can be a certain value, or it can be an indefinite value. For example, all computing units in the computing unit set that are suitable for processing model reasoning tasks can be used as the first computing unit. For another example, a predetermined number of computing units that are most suitable for processing model reasoning tasks can be selected from the computing unit set as the first computing unit.

[0065] When selecting the first computing power unit from the set of computing power units, one or more of many factors can be comprehensively considered, such as the actual performance of the computing power unit, the load situation, the degree of adaptation to the model reasoning task, the priority of the computing power unit and / or the model reasoning task, so as to select a suitable first computing power unit.

[0066] In some embodiments, the allocation score of each computing unit relative to the model inference task can be calculated for at least part of the computing units in the set of computing units. The allocation score can be regarded as a quantitative result of the computing unit under one or more of the above factors. After obtaining the allocation score of each computing unit in at least part of the computing units, a plurality of computing units with high allocation scores can be screened out from at least part of the computing units to obtain a plurality of first computing units. For example, all computing units with allocation scores greater than a predetermined threshold can be screened out to obtain a plurality of first computing units. For another example, a predetermined number of computing units with high allocation scores can also be screened out to obtain a plurality of first computing units.

[0067] At least some of the computing power units may be some or all of the computing power units in the computing power unit set. Exemplarily, based on the memory requirements and / or computing type of the model reasoning task, computing power units that meet the memory requirements of the model reasoning task and / or match the computing type of the model reasoning task can be selected from the computing power unit set to obtain at least some of the computing power units. For example, assuming that task A is a GPU-intensive task, computing power units of type GPU that meet the memory requirements of task A can be screened out from the computing power unit set to obtain at least some of the computing power units. Among them, the memory requirements and / or computing type of the model reasoning task can be determined by analyzing the model reasoning task.

[0068] Therefore, the present disclosure can first perform a preliminary screening of the computing power unit set based on the memory requirements and / or computing type of the model reasoning task, then calculate the allocation score for the computing power units obtained through the preliminary screening, and perform a secondary screening based on the allocation score to obtain multiple first computing power units.

[0069] Exemplarily, the allocation score can be positively correlated with one or more of the basic score, task fitness, load balancing factor, and priority weight. Preferably, the allocation score of the computing power unit can be calculated by combining multiple factors such as the basic score, task fitness, load balancing factor, and priority weight. The calculation formula of the allocation score can be exemplarily expressed as allocation score = basic score × task fitness × load balancing factor × priority weight.

[0070] The basic score, task fitness, load balancing factor, and priority weight are explained below.

[0071] 1) Basic score

[0072] The basic score is used to characterize the actual performance of the computing unit.

[0073] The basic score can be regarded as a numerical value that quantifies the actual performance of the computing power unit.

[0074] The actual performance may be a performance indicator that can reflect the overall capability of the computing power unit.

[0075] The actual performance of a computing unit can be expressed by two indicators: the current availability score and the current computing power score.

[0076] The current availability score, also known as the computing power availability coefficient, is used to indicate the availability of the current computing power unit. The value range of the current availability score is between 0 and 1.

[0077] The current computing power score, also known as the actual computing power equivalent value, can be used to convert different types of computing units (such as CPU and GPU) into a unified computing power measurement based on the current computing power score.

[0078] The actual performance of the computing unit can be expressed as the current availability score × the current computing power score. Therefore, the calculation formula of the basic score can be exemplarily expressed as: basic score = current availability score × current computing power score.

[0079] Exemplarily, a weighted moving average algorithm can be used to evaluate the actual performance (i.e., basic score) of the computing unit. In other words, the actual performance P(t) of the computing unit can be expressed as, P(t) = α*P(t-1) + (1-α)*P_current. Wherein, P(t) is the actual performance of the computing unit at time t, P(t-1) is the actual performance of the computing unit at time t-1, α is a smoothing factor (0<α<1), and P_current is the performance value measured (such as predicted) at the current moment (i.e., moment t).

[0080] For example, a pre-built machine learning model can be used to predict the current actual performance of the computing unit to obtain the performance value P_current measured by the computing unit at the current moment.

[0081] Taking the current availability score and the current computing power score as an example, the actual performance of the computing power unit can be represented by the two indicators of current availability score and current computing power score. The pre-built machine learning model can be used to predict the current availability score and the current computing power score of each computing power unit. The input of the machine learning model can be the status information of the computing power unit, and the output of the machine learning model can be the current availability score and the current computing power score.

[0082] The status information may include but is not limited to at least one of the following: type of computing unit; floating point computing capability; memory usage; average utilization; host bandwidth; average network I / O; average task completion time; average energy consumption; error rate; average operating temperature; current load status; and elapsed running time. The status information of the computing unit can be obtained through real-time monitoring.

[0083] The present disclosure can train multiple machine learning models, such as random forest, gradient boosting tree, and deep neural network, and select one or more machine learning models with the best model performance to estimate the current availability score and current computing power score of the power unit. The model training process is as follows.

[0084] Initial training can be performed based on historical data sets to train multiple machine learning models. The model effects (i.e., model performance) of these multiple machine learning models can then be evaluated using, but not limited to, cross-validation. Finally, the machine learning model with the best model effect performance can be selected for real-time estimation. That is, the machine learning model with the best model effect performance can be used to predict the current availability score and current computing power capacity score of each computing power unit.

[0085] The present disclosure can also continuously collect status information and computing power information of each computing unit to regularly update the machine learning model using newly collected data. For example, the above-mentioned multiple machine learning models can be updated using regularly collected new data, and the model effects of the multiple machine learning models can be re-evaluated, and the machine learning model with the best effect can be selected for real-time estimation.

[0086] The collected data may include, but is not limited to: CPU / GPU usage; memory usage; I / O status; network bandwidth utilization; historical task processing time; current load conditions; temperature and power consumption data. When constructing training samples based on the collected data, the features of the training samples can be obtained by extracting features from the collected data, and the labels of the training samples can be the current availability scores and current computing power scores obtained through manual labeling.

[0087] Exemplarily, the input features of the machine learning model may include, but are not limited to: computing unit type (GPU / CPU / NPU); historical average utilization (past 24 hours); host bandwidth of the computing unit; average network I / O of the computing unit (past 1 hour); Historical average task completion time (past 24 hours); Historical average energy consumption (past 24 hours); Historical error rate (past 24 hours); average operating temperature (past 24 hours); current load status; elapsed time (since last maintenance). The output of the machine learning model is: predicted availability score (0-1 range); predicted computing power score (relative value). During the training process: initial training can be performed using historical data from the past 30 days; the model is updated every 24 hours using newly collected data; random forest algorithms can be used for training to capture nonlinear relationships.

[0088] By introducing machine learning technology, the actual performance of the power unit can be predicted more intelligently, providing more accurate decision-making basis for task scheduling, and it can also adapt to complex and changing computing environments.

[0089] 2) Task suitability

[0090] Task fitness is used to characterize the degree of fitness between the computing unit and the model reasoning task.

[0091] The task suitability can be calculated based on the historical performance data of the computing unit.

[0092] Figure 2 is an example diagram showing historical performance data according to one embodiment of the present disclosure.

[0093] exist Figure 2In the table, u1 represents a computing unit; t1 represents a computing type; availability represents the availability score; performance represents the computing power score; avg_time represents the average task completion time (in seconds); success_rate represents the task success rate; energy_efficiency represents energy efficiency, and the value range of energy_efficiency can be between 0 and 1, and the higher the value, the better.

[0094] based on Figure 2 It can be seen that historical performance data can characterize the historical performance of the computing unit under different computing types.

[0095] Exemplarily, the task suitability can be calculated based at least on the historical performance of the computing unit under the same computing type as the model reasoning task. Thus, the task suitability takes into account the historical performance of the computing unit for a specific type of task.

[0096] Task suitability can be positively correlated with the historical performance of the computing unit under the same computing type as the model reasoning task. Different computing types have different computing characteristics. For example, computing types can include but are not limited to CPU intensive, GPU intensive, and TPU intensive.

[0097] The task fitness is positively correlated with the historical performance of the computing unit under the same computing type as the model reasoning task, and can also be negatively correlated with the best historical performance of the computing unit under all computing types. Therefore, the task fitness can reflect the gap between the performance of the computing unit under the same computing type as the model reasoning task and the best performance under the computing type it is best at, so that the calculated task fitness can more accurately reflect the degree of adaptation between the computing unit and the model reasoning task.

[0098] The historical performance may include, but is not limited to, one or more of average task completion speed, success rate, and energy efficiency. The average task completion speed may be determined based on the average task completion time. For example, the average task completion speed may be equal to the inverse of the average task completion time.

[0099] In the case where the historical performance includes multiple historical performance indicators, different historical performance indicators can be assigned certain weights based on the importance of the degree of adaptation between the computing power unit and the model reasoning task that can be reflected by the historical performance indicators, and the task adaptation can be calculated by weighted summation. The sum of all weights can be 1. For example, the historical performance indicator "average task completion speed" can be assigned a first weight (such as 0.4), the historical performance indicator "success rate" can be assigned a second weight (such as 0.2), and the historical performance indicator "energy efficiency" can be assigned a third weight (such as 0.2).

[0100] For example, the calculation formula of task fit task_fit can be expressed as:

[0101] task_fit=(time_factor*0.4+success_factor*0.4+efficiency_factor*0.2) /

[0102] max((1 / H[u][task_type)]['avg_time']*0.4+

[0103] H[u][task_type]['success_rate']*0.4+

[0104] H[u][task_type)]['energy_efficiency']*0.2)

[0105] Among them, time_factor is the average task completion speed of the computing unit under the same computing type as the model reasoning task; success_factor is the success rate of the computing unit under the same computing type as the model reasoning task; efficiency_factor is the energy efficiency of the computing unit under the same computing type as the model reasoning task; (1 / H[u][task_type)]['avg_time'] is the average task completion speed of the computing unit under the computing type of task_type; H[u][task_type]['success_rate'] is the success rate of the computing unit under the computing type of task_type; H[u][task_type)]['energy_efficiency'] is the energy efficiency of the computing unit under the computing type of task_type. The value range of task_type can be all computing types. The max() part in the formula is used to represent the weighted summation result of the best historical performance of the computing unit for different historical performance indicators under all computing types.

[0106] After the first computing unit completes the calculation, the historical performance of the first computing unit may also be updated. For example, the historical performance of the first computing unit may be updated based on the performance of the computing task performed by the first computing unit this time (such as task completion time, whether it is successfully completed, and energy efficiency).

[0107] 3) Load balancing factor

[0108] The load balance factor is used to characterize the degree of load balance of the computing power unit.

[0109] By calculating the allocation score based on the load balancing factor, it is possible to avoid screening out the overloaded first computing unit.

[0110] The load balancing factor may be negatively correlated with the current load of the computing unit. That is, the greater the current load of the computing unit, the smaller the load balancing factor. In order to accurately assess the load of the computing unit, the load balancing factor may be negatively correlated with the current load of the computing unit and positively correlated with the maximum load of the computing unit.

[0111] Exemplarily, the calculation formula of the load balancing factor load_factor can be expressed as:

[0112] load_factor=1-(current_load(unit) / max_load(unit)), where current_load(unit) represents the current load of the computing power unit, and max_load(unit) represents the maximum load of the computing power unit.

[0113] 4) Priority Weight

[0114] The priority weight is used to represent the priority weight of a computing unit and / or a model reasoning task.

[0115] The weight of the priority of the computing unit can ensure that high-priority computing units are selected for model reasoning tasks first. The weight of the priority of the model reasoning task can ensure that high-priority tasks get better computing resources first.

[0116] The priority weight may be positively correlated with the priority of the model inference task and negatively correlated with the maximum priority of all currently pending model inference tasks. And / or the priority weight may be positively correlated with the priority of the computing power unit.

[0117] In step S120, the model required for executing the model reasoning task is split into multiple sub-models.

[0118] Model parallelism can be used to split a model into multiple sub-models. The specific splitting principle is a mature technology in this field. For example, the model can be decomposed hierarchically, and the parallelism of the model can be taken into account during the decomposition process to split the model into multiple sub-models that can be executed in parallel. Each sub-model corresponds to a part of the parameters of the model.

[0119] In step S130, with the goal of at least making the task completion times of the multiple first computing units consistent or substantially consistent, the multiple sub-models are assigned to the multiple first computing units so that the first computing units perform the computing tasks corresponding to the sub-models assigned to them.

[0120] By determining the allocation strategies of multiple sub-models with the goal of at least making the task completion times of multiple first computing units consistent or substantially consistent, multiple first computing units can complete the computing tasks assigned to them synchronously or substantially synchronously, thereby reducing the overall waiting time caused by the delay of a single computing unit.

[0121] In some embodiments, multiple sub-models may be assigned to the first computing units with the goal of at least minimizing the overall task completion time difference. The overall task completion time difference is used to characterize the difference between the first task completion time of the first computing unit that completes the calculation first and the second task completion time of the first computing unit that completes the calculation last. In this way, it can be ensured that each computing unit completes the reasoning calculation in the most consistent time possible.

[0122] Exemplarily, the present disclosure may also use a pre-trained machine learning model to predict the computing time of each first computing unit for each sub-model, and the computing time is used to characterize the time required for the first computing unit to perform the computing task corresponding to the sub-model. Based on the computing time of each first computing unit for each sub-model, multiple sub-models are assigned to multiple first computing units. Among them, the total computing time of each first computing unit to which a sub-model is assigned is consistent or substantially consistent, and the total computing time is used to characterize the sum of the computing time of the first computing unit for all sub-models assigned to it. Thus, allocation and management can be performed within a predetermined time frame according to the actual capabilities of the computing units to ensure completion on time.

[0123] Exemplarily, when allocating multiple sub-models to multiple first computing units, in addition to making the task completion time of multiple first computing units consistent or substantially consistent as the goal, the overall completion time of the model reasoning task can also be taken into account. That is, the allocation strategy of multiple sub-models can be determined to allocate multiple sub-models to multiple first computing units with the goal of making the task completion time of multiple first computing units consistent or substantially consistent and reducing the overall completion time of the model reasoning task as much as possible. In this way, while ensuring that each computing unit completes the reasoning calculation in the most consistent time possible, the overall delay of the model reasoning task can also be reduced.

[0124] In the actual solution process, based on the above objectives, a dynamic programming (DP) algorithm can be used but not limited to be solved to determine the allocation strategy of multiple sub-models. Dynamic programming is a branch of operations research and is a process for optimizing the decision-making process. Dynamic programming algorithms are usually used to solve problems with certain optimal properties, so they can be used in the present disclosure to solve the allocation strategy problem of multiple sub-modules.

[0125] After the sub-model is assigned to the first computing unit, the status information of the first computing unit can be updated, and / or the performance of the first computing unit can be re-evaluated using the machine learning model. In this way, it can be ensured that the computing units are in the latest state during subsequent assignments. When updating the status information of the first computing unit, the current load and remaining memory of the first computing unit can be updated based on the complexity and memory usage of the computing tasks corresponding to the sub-model assigned to the first computing unit.

[0126] In step S140, based on the calculation results of the multiple first computing units, the inference result of the model inference task is obtained.

[0127] The calculation results of multiple first computing units can be summarized to obtain the reasoning result of the model reasoning task.

[0128] The execution process of the model reasoning task may require one or more rounds of reasoning. Each round of reasoning requires calling the model once. That is, each round of reasoning requires these multiple first computing units to perform corresponding computing tasks based on the sub-models assigned to them. Therefore, the calculation results of multiple first computing units in the same reasoning round can be summarized to obtain the reasoning results of the round. Similarly, the results of multiple rounds of reasoning are summarized to obtain the reasoning results of the model reasoning task.

[0129] Taking the model reasoning task as a text generation task as an example, the words generated in the current round can be obtained based on the calculation results of the multiple first computing units in each round of reasoning process, and then the words obtained in multiple rounds of reasoning processes can be combined in chronological order to obtain the text generation result corresponding to the text generation task.

[0130] In some embodiments, the performance changes of the computing power unit can also be monitored (i.e., observed). Exemplarily, the performance changes of the computing power unit can be directly analyzed based on the changes in the status information of the computing power unit; and / or, the performance changes of the computing power unit can also be determined based on the changes in the results of predicting the actual performance of the computing power unit using a machine learning model. After determining the performance changes of the computing power unit, a dynamic adjustment strategy that matches the performance changes of the computing power unit can be executed for the computing power unit. The dynamic adjustment strategy may include, but is not limited to, at least one of the following: adjusting the priority of the computing power unit in the allocation process; adjusting the complex task allocation ratio of the computing power unit; adjusting the energy management strategy of the computing power unit; adjusting the cooling resource allocation strategy of the computing power unit; adjusting the maintenance and inspection strategy of the computing power unit.

[0131] For example, for the computing unit NVIDIA V100 GPU, its performance changes can be observed after the system runs for a week. For example, the current status information of NVIDIA V100 GPU is:

[0132] Type: GPU;

[0133] Historical average utilization rate: 85% (↑10%);

[0134] Host bandwidth of computing unit: 10G;

[0135] The network I / O of the computing unit (in the past hour) is 3M / s;

[0136] Historical average task completion time: 1.2 seconds / task (↑0.3 seconds);

[0137] Historical average energy consumption: 250W (↑30W);

[0138] Historical error rate: 0.5% (↑0.3%);

[0139] Average operating temperature: 75℃(↑5℃);

[0140] Current load status: 90%;

[0141] Elapsed running time: 168 hours.

[0142] The above status information is input into the pre-built machine learning model to obtain the availability score and computing power score output by the machine learning model, specifically: availability score: 0.85 (↓0.1); computing power score: 0.92 (↓0.05).

[0143] Based on the above status information and the performance prediction information output by the model, the following operations can be performed on the GPU: lower the priority of the GPU in task allocation; increase the cooling resource allocation to the GPU; and schedule a maintenance inspection during the next off-peak period.

[0144] For another example, after the system has been running for a week, the current status information of HUAWEI 910B is:

[0145] Type: NPU;

[0146] Historical average utilization rate: 60% (↓15%);

[0147] Host bandwidth of computing unit: 10G;

[0148] Network I / O of computing unit (last hour): 1M / s;

[0149] Historical average task completion time: 2 seconds / task (↓0.5 seconds);

[0150] Historical average energy consumption: 260W (↓10W);

[0151] Historical error rate: 0.1% (no change);

[0152] Average operating temperature: 65℃(↓3℃);

[0153] Current load status: 55%;

[0154] Elapsed running time: 168 hours.

[0155] The above status information is input into the pre-built machine learning model to obtain the availability score and computing power score output by the machine learning model, specifically: availability score: 0.98 (↑0.05); computing power score: 1.08 (↑0.06).

[0156] Based on the above status information and the performance prediction information output by the model, the following operations can be performed on the NPU: increase the priority of the NPU in task allocation; increase the proportion of complex tasks assigned to the NPU; adjust the energy management strategy to allow a higher performance mode.

[0157] As a result, the performance changes of each computing unit can be monitored in real time, and adjustment strategies that match the performance changes of each computing unit can be implemented for each computing unit. By re-evaluating unit performance through continuously updated machine learning models, it is possible to quickly adapt to changes in unit performance, such as thermal throttling or hardware problems.

[0158] Comparing the overall system performance one week before and after the adjustment, we found that the average task completion time was reduced by 8%, energy efficiency was improved by 12%, and system stability was improved by 5%. Through this case, this paper shows how to use machine learning technology to evaluate the availability and capacity of computing units in real time and dynamically adjust the management strategy of computing units. This adaptive mechanism enables the system to maintain high efficiency and stability in long-term operation, while providing data support for hardware resource management.

[0159] In some embodiments, the load of each computing unit can also be monitored. When load imbalance is detected, task migration is triggered. Task migration takes into account migration costs and performance benefits, and is only executed when the benefits are greater than the costs.

[0160] Specifically, it can be determined whether the difference in the computational progress of different first computing units is greater than a first threshold. If the difference in the computational progress of different first computing units is greater than or equal to the first threshold (e.g., 5%), it is analyzed whether the performance benefit of assigning the unexecuted sub-model in the first computing unit with a relatively lagging computational progress to the first computing unit with a relatively advanced computational progress is greater than the migration cost. If the performance benefit is greater than the migration cost, the unexecuted sub-model in the first computing unit with a relatively lagging computational progress is assigned to the first computing unit with a relatively advanced computational progress.

[0161] For example, it is possible to periodically check whether the difference between the calculation progress of different first computing units is greater than the first threshold, or to check whether the difference between the calculation progress of different first computing units is greater than the first threshold each time a first computing unit completes a certain progress ratio (such as 25%, 50%, 75%). If the difference between the calculation progress of different first computing units is less than the first threshold, no further adjustment is required; otherwise, the above migration process can be executed.

[0162] The scheduling method disclosed in the present invention can be implemented as a scheduling system, which can also be called a multi-type heterogeneous computing power adaptive scheduling system. Figure 3 It is a schematic diagram showing the overall architecture of a scheduling system according to a disclosed embodiment.

[0163] like Figure 3 As shown, the scheduling system as a whole includes three parts: task management, adaptive task allocation and resource management.

[0164] The task management part involves task reception and task analysis. Task reception refers to receiving model reasoning tasks submitted by users through the client. Task analysis refers to analyzing the received model reasoning tasks to determine the priority, complexity, and estimated computing resources required for the model reasoning tasks. Taking the text generation task as an example, the received model reasoning task may include an input prompt and the required output length.

[0165] Adaptive task allocation mainly involves scheduling optimizer, task scheduling, executor and machine learning estimation.

[0166] Machine learning estimation means that the computing power availability coefficient (i.e., current availability score) and actual computing power equivalent value (i.e., current computing power capacity score) of a computing unit can be estimated in real time using machine learning algorithms.

[0167] Task scheduling refers to assigning tasks to appropriate computing units based on the performance evaluation results of computing units estimated by machine learning. For example, the following strategies can be used for task scheduling: preliminarily screen suitable computing units based on task types (such as CPU-intensive / GPU-intensive); consider the current load and performance evaluation value of the computing unit; use priority queues to maintain available computing units, with priorities based on performance evaluation values ​​and current loads; and assign multiple sub-models obtained by splitting the model to multiple computing units.

[0168] Exemplarily, a dynamic adaptive allocation algorithm can be used for task scheduling. The input of the dynamic adaptive allocation algorithm may include the current set of tasks to be processed T, the set of computing units U, and the historical performance data H. Task set T{t1, t2, ..., tn}. Each task ti in the task set T may include an estimated computing complexity complexity(ti), a memory requirement memory_req(ti), and a priority priority(ti). Computing unit set U={u1, u2, ..., um}. Each computing unit uj in the computing unit set U may include a type (GPU / CPU / TPU), a current availability score availability(uj), a current computing power score performance(uj), and current free memory free_memory(uj). The historical performance data H(uj, task_type) is used to characterize the historical performance of the computing unit uj in processing tasks of a specific type. The dynamic adaptive allocation algorithm can be combined with the above. Figure 1 The described scheduling method selects a suitable first computing unit for each task in the task set T.

[0169] The scheduling optimizer refers to a system that can use a dynamic programming algorithm to optimize task allocation to minimize the overall completion time difference. For example, the scheduling optimizer can be used to determine the allocation strategy between multiple sub-models and multiple first computing units.

[0170] The executor can instruct the computing unit to perform the corresponding computing task according to the task scheduling result, and can summarize the calculation results of the computing unit to obtain the inference result of the model inference task.

[0171] The resource management part mainly involves resource registration and discovery, resource monitoring and evaluation, and resource scheduling.

[0172] Resource registration and discovery means that newly discovered computing resources can be registered into the system so that the system can automatically evaluate their performance and take them into consideration for scheduling.

[0173] Resource monitoring and evaluation refers to the ability to evaluate the actual performance and load of each computing unit in real time based on the performance data and status information collected from each computing unit. Therefore, the scheduling system may also include a data acquisition module responsible for collecting the performance data and status information of each computing unit.

[0174] Resource scheduling refers to the ability to dynamically adjust computing units based on the resource monitoring and evaluation of each computing unit. For example, you can dynamically adjust the load balancing strategy; update the weight coefficient in the task allocation algorithm; adjust the maintenance plan to prioritize units with degraded performance; and generate performance trend reports for administrators to assist in hardware upgrade decisions.

[0175] Taking a large language model reasoning task as an example, the present disclosure can also be implemented as a heterogeneous computing environment adaptive scheduling system for large model reasoning tasks. The scheduling process of a large language model reasoning task is exemplified below.

[0176] Suppose you receive a text generation inference task using a large language model like GPT-3 with 175B parameters. The task requires generating an article of 500 words. Currently available heterogeneous computing resources include 4 NVIDIA A100 GPUs, 2 AMD EPYC CPUs, and 4 NVIDIA A10 GPUs.

[0177] The current status of each computing unit can be collected in real time, including load, memory usage, temperature, etc.

[0178] Large models can be decomposed hierarchically to take into account model parallelism: a 175B parameter model is divided into 35 5B parameter sub-models; each sub-model is responsible for processing a specific model layer or component.

[0179] Use the machine learning estimation module to predict the time it takes for each computing unit to process each sub-model. For example, the estimated processing time for A100 GPU for each 5B parameter sub-model is 2 seconds; the estimated processing time for AMD EPYC CPU for each 5B parameter sub-model is 20 seconds; and the estimated processing time for A10 GPU for each 5B parameter sub-model is 8 seconds.

[0180] Since the AMD EPYC CPU takes a long time to process each 5B parameter sub-model, and the AMD EPYC CPU is not suitable for processing large model inference tasks. Therefore, a dynamic programming algorithm can be used for 4 A100 GPUs and 4 A10 GPUs to minimize the overall completion time difference and determine the allocation strategy for the multiple sub-models obtained by splitting. The allocation results are: 31 sub-models are allocated to 4 A100 GPUs (each processing 7-8 sub-models); 4 sub-models are allocated to the A10 GPU.

[0181] Start the inference task and activate the real-time monitoring mode at the same time. Check the progress of each computing unit every 100ms and record the completion time of each sub-model. If the monitoring finds that the temperature of one A100 GPU rises and the performance decreases, the two unstarted subtasks (i.e., sub-models) on the GPU can be immediately reallocated; for example, the two unstarted subtasks can be transferred to the other two A100 GPUs with lighter loads.

[0182] Synchronous checkpoints can be set when each sub-model is 25%, 50%, and 75% complete. For each synchronous checkpoint, the progress of each unit can be compared. If the maximum deviation is found to be within an acceptable range (<5%), no further adjustment is required and execution continues; otherwise, task migration can be triggered.

[0183] As the sub-models are completed, the system begins to summarize the results. You can use pipelined result processing to summarize and output at the same time. After all rounds of calculations are completed, the system performs the final result integration to ensure the coherence and completeness of the 500-word article.

[0184] The total time taken was about 14 seconds (about 5 times faster than single device serial processing).

[0185] The detailed execution data of this task can be recorded so that the machine learning module can use this data to update the performance prediction model. The scheduling algorithm can fine-tune parameters based on the new data and optimize the scheduling strategy for similar tasks in the future.

[0186] This detailed case analysis clearly demonstrates how the scheduling system can efficiently handle large model inference tasks, especially how to achieve synchronous scheduling and dynamic optimization of heterogeneous computing power.

[0187] With the development of artificial intelligence technology, the demand for computing resources for large model reasoning tasks is growing. Traditional homogeneous computing environments are difficult to meet this demand, and although heterogeneous computing environments provide higher flexibility and computing efficiency, they face huge challenges in task scheduling. Especially in large model reasoning scenarios, how to ensure that each computing unit completes the task synchronously to minimize overall latency has become an urgent problem to be solved.

[0188] The present disclosure can be implemented as a multi-type heterogeneous computing power adaptive synchronous scheduling scheme for large model reasoning. The scheme can dynamically evaluate the actual performance of each computing unit in a heterogeneous environment, perform intelligent task allocation, and ensure that each computing unit completes the reasoning calculation in as consistent a time as possible, thereby significantly improving the efficiency and response speed of large model reasoning. In addition, the present disclosure not only allocates tasks based on the performance of the computing unit, but also takes into account the priority and completion time consistency of the task, ensuring that each unit can complete the assigned reasoning task almost synchronously.

[0189] The present disclosure can be flexibly adjusted according to real-time performance evaluation and task characteristics, thereby achieving efficient resource utilization in a heterogeneous environment. This process takes into account the current state (availability, performance) of each unit and its historical performance in handling similar tasks to make the best allocation decision. Historical performance data can be integrated into a dynamic adaptive allocation algorithm, enabling the system to make smarter decisions based on past experience. This approach takes into account not only the current state, but also the long-term performance, so as to find the best match between different types of tasks and computing power units with different characteristics.

[0190] The present disclosure has at least the following advantages: improving resource utilization, by accurately evaluating the actual capabilities of each computing unit, the system can allocate tasks more effectively and reduce resource waste; optimizing performance, the adaptive scheduling mechanism ensures that tasks are assigned to the most appropriate computing unit, thereby improving overall computing performance; flexibility, the system can adapt to computing devices of different types and architectures, providing enterprises with greater flexibility in hardware selection; scalability, as new computing devices are added, the system can automatically evaluate their performance and take them into consideration in scheduling; intelligence, by introducing machine learning technology, the system can more intelligently predict the actual performance of computing units and adapt to complex and changing computing environments; improving reasoning efficiency, by ensuring that each computing unit completes tasks synchronously, the overall delay of large model reasoning is significantly reduced, and the response speed of the system is improved; optimizing resource utilization, the synchronous scheduling mechanism ensures the full utilization of various computing resources in a heterogeneous environment, and avoids the overall waiting time caused by delays in a single node.

[0191] The scheduling method disclosed in the present invention can also be implemented as a scheduling device. The functional units of the scheduling device can be implemented by hardware, software, or a combination of hardware and software that implements the principles of the present invention. The following is a brief description of the functional units that the scheduling device can have and the operations that each functional unit can perform. For the details involved, please refer to the relevant description above, which will not be repeated here. It can be understood by those skilled in the art that the functional units described in this article can be combined or divided into sub-units to implement the principles of the above invention. Therefore, the description herein can support any possible combination, or division, or further limitation of the functional units described herein.

[0192] Figure 4 FIG. 1 is a schematic diagram showing the structure of a scheduling device according to an embodiment of the present disclosure. Figure 4 The scheduling device 400 may include a screening module 410 , a splitting module 420 , an allocation module 430 and a processing module 440 .

[0193] The screening module 410 is used to screen out multiple first computing units from the computing unit set for the model reasoning task. The splitting module 420 is used to split the model required to perform the model reasoning task into multiple sub-models. The allocation module 430 is used to allocate the multiple sub-models to the multiple first computing units, at least with the goal of making the task completion time of the multiple first computing units consistent or basically consistent, so that the first computing units perform the computing tasks corresponding to the sub-models allocated to them. The processing module 440 is used to obtain the inference result of the model reasoning task based on the calculation results of the multiple first computing units.

[0194] In some embodiments, the allocation module 430 can allocate the multiple sub-models to the multiple first computing units with the goal of at least minimizing the overall task completion time difference, where the overall task completion time difference is used to characterize the difference between the first task completion time of the first operator unit that completes the calculation first and the second task completion time of the first operator unit that completes the calculation last.

[0195] In some embodiments, the scheduling device 400 may further include a first prediction module. The first prediction module is used to predict the calculation time of each of the first computing units for each of the sub-models, and the calculation time is used to characterize the time required for the first computing unit to perform the calculation task corresponding to the sub-model. The allocation module 430 may allocate the multiple sub-models to the multiple first computing units based on the calculation time of each of the first computing units for each of the sub-models, wherein the total calculation time of each of the first computing units to which the sub-models are allocated is consistent or substantially consistent, and the total calculation time is used to characterize the sum of the calculation times of the first computing unit for all the sub-models allocated to it.

[0196] In some embodiments, the screening module 410 may include a calculation module and a screening submodule. The calculation module is used to calculate the allocation score of each computing unit relative to the model reasoning task for at least part of the computing units in the computing unit set. The screening submodule is used to screen out a plurality of computing units with high allocation scores from at least part of the computing units to obtain the plurality of first computing units.

[0197] Exemplarily, the scheduling device 400 may further include a selection module. The selection module is used to select a computing unit that meets the memory requirement and / or matches the computing type from the computing unit set based on the memory requirement and / or computing type of the model reasoning task, to obtain at least part of the computing units.

[0198] Exemplarily, the allocation score can be positively correlated with one or more of the basic score, task fitness, load balancing factor, and priority weight, respectively, wherein the basic score is used to characterize the actual performance of the computing unit, the task fitness is used to characterize the degree of adaptation between the computing unit and the model reasoning task, the load balancing factor is used to characterize the load balancing degree of the computing unit, and the priority weight is used to characterize the weight of the priority of the computing unit and / or the model reasoning task. For the basic score, task fitness, load balancing factor, and priority weight, please refer to the relevant description above.

[0199] Exemplarily, the basic score is positively correlated with the current availability score and the current computing power score, respectively. The scheduling device 400 may further include a second prediction module. The second prediction module is used to predict the current availability score and the current computing power score of each computing power unit using a pre-built machine learning model, the input of the machine learning model is the status information of the computing power unit, and the output of the machine learning model is the current availability score and the current computing power score. Exemplarily, the scheduling device 400 may further include a training module and an evaluation module. The training module is used to train multiple machine learning models based on historical data sets. The evaluation module is used to evaluate the model effects of the multiple machine learning models. The second prediction module uses the machine learning model with the best model effect performance to predict the current availability score and the current computing power score of each computing power unit.

[0200] Exemplarily, the scheduling device 400 may further include a first updating module. The first updating module is used to update the historical performance of the first computing unit after the first computing unit completes the calculation.

[0201] In some embodiments, the scheduling device 400 may further include a monitoring module and an adjustment module. The monitoring module is used to monitor the performance changes of the computing unit. The adjustment module is used to execute a dynamic adjustment strategy matching the performance changes of the computing unit.

[0202] In some embodiments, the scheduling device 400 may further include a second updating module. The second updating module is used to update the state information of the first computing unit after the sub-model is assigned to the first computing unit, and / or re-evaluate the performance of the first computing unit using a machine learning model.

[0203] In some embodiments, the scheduling device 400 may also include a judgment module, an analysis module, and a migration module. The judgment module is used to judge whether the difference between the calculation progress of different first computing power units is greater than a first threshold. The analysis module is used to analyze whether the performance benefit generated by allocating the unexecuted sub-model in the first computing power unit with relatively lagging calculation progress to the first computing power unit with relatively leading calculation progress is greater than the migration cost if the difference between the calculation progress of different first computing power units is greater than or equal to the first threshold. The migration module is used to allocate the unexecuted sub-model in the first computing power unit with relatively lagging calculation progress to the first computing power unit with relatively leading calculation progress if the performance benefit is greater than the migration cost.

[0204] Figure 5 A schematic diagram of the structure of a computing device that can be used to implement the above scheduling method according to an embodiment of the present disclosure is shown.

[0205] See also Figure 5 , the computing device 500 includes a memory 510 and a processor 520 .

[0206] The processor 520 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 520 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 520 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0207] The memory 510 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 520 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at runtime. In addition, the memory 510 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 510 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0208] The memory 510 stores executable codes, and when the executable codes are processed by the processor 520 , the processor 520 can execute the scheduling method mentioned above.

[0209] The scheduling method, apparatus and device according to the present disclosure have been described above in detail with reference to the accompanying drawings.

[0210] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0211] Alternatively, the present disclosure may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present disclosure.

[0212] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0213] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the system and method according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0214] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A scheduling method, characterized in that: include: For the model inference task, multiple first computing units are selected from the computing unit set; Splitting the model required to perform the model reasoning task into multiple sub-models; At least with the goal of making the task completion times of the plurality of first computing units consistent or substantially consistent, the plurality of sub-models are assigned to the plurality of first computing units, so that the first computing units perform the computing tasks corresponding to the sub-models assigned thereto; Based on the calculation results of the multiple first computing units, the reasoning result of the model reasoning task is obtained.

2. The method according to claim 1, characterized in that At least with the goal of making the task completion times of the plurality of first computing units consistent or substantially consistent, the plurality of sub-models are allocated to the plurality of first computing units, including: The multiple sub-models are assigned to the multiple first computing units with the goal of at least minimizing the overall task completion time difference, and the overall task completion time difference is used to characterize the difference between the first task completion time of the first operator unit that completes the calculation first and the second task completion time of the first operator unit that completes the calculation last.

3. The method according to claim 1, characterized in that Also includes: Predicting the computing time of each of the first computing units for each of the sub-models, where the computing time is used to represent the time required for the first computing unit to perform the computing task corresponding to the sub-model; At least with the goal of making the task completion time of the multiple first computing units consistent or substantially consistent, the multiple sub-models are allocated to the multiple first computing units, including: based on the calculation time of each of the first computing units for each of the sub-models, the multiple sub-models are allocated to the multiple first computing units, wherein the total calculation time of each first computing unit to which the sub-model is allocated is consistent or substantially consistent, and the total calculation time is used to represent the sum of the calculation time of the first computing unit for all the sub-models allocated to it.

4. The method according to claim 1, characterized in that: For the model inference task, multiple first computing units are selected from the computing unit set, including: For at least part of the computing units in the computing unit set, calculating the allocation score of each of the computing units relative to the model reasoning task; A plurality of computing units with high allocation scores are selected from at least part of the computing units to obtain the plurality of first computing units.

5. The method according to claim 1, characterized in that Also includes: Monitoring performance changes of the computing unit; A dynamic adjustment strategy matching the performance change of the computing power unit is executed.

6. The method according to any one of claims 1 to 5, characterized in that The model reasoning task is a text generation task; Based on the calculation results of the multiple first computing units, the inference result of the model inference task is obtained, including: Based on the calculation results of the plurality of first computing units in each round of reasoning process, obtaining the words generated in the current round; The words obtained from multiple rounds of reasoning processes are combined in chronological order to obtain a text generation result corresponding to the text generation task.

7. A scheduling device, characterized in that: include: A screening module, used for screening out a plurality of first computing units from a set of computing units for a model inference task; A splitting module, used to split the model required for performing the model reasoning task into multiple sub-models; an allocation module, configured to allocate the plurality of sub-models to the plurality of first computing units at least with the goal of making the task completion times of the plurality of first computing units consistent or substantially consistent, so that the first computing units perform the computing tasks corresponding to the sub-models allocated thereto; A processing module is used to obtain the inference result of the model inference task based on the calculation results of the multiple first computing power units.

8. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 6.

9. A computer program product, comprising executable codes, which, when executed by a processor of an electronic device, causes the processor to perform the method according to any one of claims 1 to 6.

10. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 6.

Images