Task scheduling method and device for artificial intelligence algorithm test task, computer equipment, storage medium and program product

Abstract

The invention provides a task scheduling method and device for an artificial intelligence algorithm test task, computer equipment, a storage medium and a program product, and relates to the technical field of computers and artificial intelligence, and the implementation scheme is as follows: obtaining respective task feature information of a plurality of reference test tasks; constructing a first prediction model and a second prediction model based on the task feature information; obtaining constraint information for executing the current test task and degradable information for satisfying the constraint information; based on the first prediction model, the second prediction model and the constraint information, determining whether a first candidate scheduling result which simultaneously meets the first sub-constraint information and the second sub-constraint information and is used for executing the current test task exists or not; and in response to determining that the first candidate scheduling result does not exist, determining a second candidate scheduling result used for executing the current test task based on the downgradable information, so that multi-target-oriented test task scheduling and collaborative optimization of test efficiency, cost control and energy conservation and emission reduction can be realized.

Description

Technical Field

[0001] This disclosure relates to the fields of computer and artificial intelligence technology, particularly to the fields of artificial intelligence algorithm testing, test task scheduling, energy-saving optimization and evaluation, and especially to a task scheduling method, apparatus, computer equipment, computer-readable storage medium and computer program product for artificial intelligence algorithm testing tasks. Background Technology

[0002] With the development of artificial intelligence technology, the scale of various large-scale models continues to expand, and the frequency of algorithm iteration is accelerating, making the workload of related algorithm testing increasingly arduous. Coupled with the ever-growing size of test clusters, this puts significant pressure on testing efficiency. Common test task scheduling strategies generally focus more on efficiency metrics, aiming to complete test tasks faster or make better use of resource allocation. However, as energy consumption and cost issues gradually attract attention, the industry is exploring ways to integrate energy conservation and cost optimization with test engineering semantics. There is an urgent need for a multi-objective-oriented test task scheduling strategy to adapt to the dual challenges of efficiency and green computing. Summary of the Invention

[0003] This disclosure provides a task scheduling method, apparatus, computer device, computer-readable storage medium, and computer program product for testing artificial intelligence algorithms.

[0004] According to one aspect of this disclosure, a task scheduling method for testing artificial intelligence algorithms is provided, comprising: acquiring task feature information of multiple reference test tasks, wherein the multiple reference test tasks include at least one task type, and the task feature information includes first reference information associated with the task metadata and test dataset of the reference test tasks, second reference information associated with the resource configuration data of the reference test tasks, and third reference information associated with the energy consumption data and multi-dimensional cost data of the reference test tasks; constructing a first prediction model and a second prediction model for each task type based on at least one task feature information corresponding to that task type, wherein the first prediction model is constructed based on reference information related to energy consumption data in the first, second, and third reference information, for use in scheduling current test tasks with the corresponding task types. Energy consumption data is used for prediction. A second prediction model is constructed based on reference information related to multi-dimensional cost data from the first, second, and third reference information to predict the multi-dimensional cost data of the current test task. Constraint information for executing the current test task and degradable information for satisfying the constraints are obtained. The constraint information includes first sub-constraint information associated with energy consumption data and multi-dimensional cost data, and second sub-constraint information associated with test effect data of the current test task. Based on the first prediction model, the second prediction model, and the constraint information, it is determined whether there is a first candidate scheduling result for executing the current test task that simultaneously satisfies the first and second sub-constraint information. In response to determining that there is no first candidate scheduling result, a second candidate scheduling result for executing the current test task is determined based on the degradable information.

[0005] In some embodiments, each of the first candidate scheduling result and the second candidate scheduling result includes a first sub-scheduling result associated with the scheduling strategy of the current test task, a second sub-scheduling result associated with the resource configuration data of the current test task, and a third sub-scheduling result associated with the energy consumption data and multi-dimensional cost data of the current test task.

[0006] In some embodiments, the first sub-scheduling result includes at least one of the following: the batch into which the current test task is divided, the test priority order of the test dataset, the test precision mode, the test review ratio, the test period, and the test concurrency; and the second sub-scheduling result includes the computing resource specifications used for the current test task.

[0007] In some embodiments, the degradability information indicates the permissible range within which the first sub-scheduling result and the second sub-scheduling result can be adjusted, provided that the first sub-constraint information and the second sub-constraint information are satisfied.

[0008] In some embodiments, the task scheduling method further includes: during the execution of the current test task based on the first candidate scheduling result or the second candidate scheduling result, determining whether either the monitoring data of energy consumption data or the monitoring data of multi-dimensional cost data is lower than the corresponding predicted data to reach a first predetermined threshold; and in response to determining that either the monitoring data of energy consumption data or the monitoring data of multi-dimensional cost data is lower than the corresponding predicted data to reach the first predetermined threshold, optimizing the first candidate scheduling result or the second candidate scheduling result based on the allowable range that can be adjusted in the first sub-scheduling result and the second sub-scheduling result.

[0009] In some embodiments, the task scheduling method further includes: during the execution of the current test task based on the first candidate scheduling result or the second candidate scheduling result, determining whether either the monitoring data of energy consumption data or the monitoring data of multi-dimensional cost data exceeds the corresponding predicted data to reach a second predetermined threshold; and in response to determining that either the monitoring data of energy consumption data or the monitoring data of multi-dimensional cost data exceeds the corresponding predicted data to reach the second predetermined threshold, correcting the first candidate scheduling result or the second candidate scheduling result based on the allowable range that can be adjusted in the first sub-scheduling result and the second sub-scheduling result.

[0010] In some embodiments, the first candidate scheduling result and the second candidate scheduling result are obtained through iterative optimization based on a heuristic or hierarchical strategy.

[0011] In some embodiments, the first and second prediction models are constructed based on linear regression modeling, decision tree modeling, or piecewise linear modeling.

[0012] According to one aspect of this disclosure, a task scheduling apparatus for testing artificial intelligence algorithms is provided, comprising: a modeling information acquisition module configured to acquire task feature information of multiple reference test tasks, wherein the multiple reference test tasks include at least one task type, and the task feature information includes first reference information associated with the task metadata and test dataset of the reference test tasks, second reference information associated with the resource configuration data of the reference test tasks, and third reference information associated with the energy consumption data and multi-dimensional cost data of the reference test tasks; and a prediction model construction module configured to construct a first prediction model and a second prediction model for each task type based on at least one task feature information corresponding to that task type, wherein the first prediction model is constructed based on reference information related to energy consumption data in the first, second, and third reference information, for predicting the energy consumption data of the current test task with the corresponding task type. The test involves a second prediction model constructed based on reference information related to multi-dimensional cost data from the first, second, and third reference information, used to predict the multi-dimensional cost data of the current test task; a constraint information acquisition module configured to acquire constraint information for executing the current test task, and degradable information for satisfying the constraint information, wherein the constraint information includes first sub-constraint information associated with energy consumption data and multi-dimensional cost data, and second sub-constraint information associated with the test effect data of the current test task; a first scheduling determination module configured to determine, based on the first prediction model, the second prediction model, and the constraint information, whether there exists a first candidate scheduling result for executing the current test task that simultaneously satisfies the first and second sub-constraint information; and a second scheduling determination module configured to, in response to determining that there is no first candidate scheduling result, determine a second candidate scheduling result for executing the current test task based on the degradable information.

[0013] According to another aspect of this disclosure, a computer device is provided, comprising: at least one processor; and a memory having a computer program stored thereon, wherein when executed by the at least one processor, the computer program causes the at least one processor to perform the methods provided above in this disclosure.

[0014] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0015] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0016] According to one or more embodiments of this disclosure, it is possible to meet given multi-dimensional cost requirements while ensuring test effectiveness, realize multi-objective-oriented test task scheduling, and thereby achieve synergistic optimization of test efficiency, cost control and energy conservation and emission reduction.

[0017] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0018] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 This is a flowchart illustrating a task scheduling method for an artificial intelligence algorithm testing task according to an embodiment of the present disclosure.

[0020] Figure 2 This is a schematic diagram illustrating the first candidate scheduling result and the second candidate scheduling result according to an embodiment of the present disclosure.

[0021] Figure 3 This is a flowchart illustrating a task scheduling method for an artificial intelligence algorithm testing task according to another embodiment of the present disclosure.

[0022] Figure 4 This is a block diagram illustrating a task scheduling apparatus for testing artificial intelligence algorithms according to an embodiment of the present disclosure.

[0023] Figure 5 An example computer device is shown in which any of the embodiments described herein may be implemented. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0026] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0027] As mentioned earlier, in traditional test task scheduling strategies, cluster scheduling and test platform pipelines focus more on efficiency metrics, with relatively singular objectives, typically aiming to complete test tasks faster or make better use of resource allocation. In recent years, although energy saving and cost optimization have been incorporated into some testing scenarios, there is still a lack of systematically providing dynamic and flexible test task scheduling strategies to adapt to multi-objective optimization approaches, including efficiency and green computing.

[0028] Therefore, embodiments of this disclosure provide a task scheduling method for testing artificial intelligence algorithms. Based on a task profile model constructed from historical test data, the method uses adjustable dynamic strategies to ensure test effectiveness (such as coverage of key test scenarios, regression test completion rate of key test versions, and on-time completion of test tasks) while meeting given multi-dimensional cost requirements (such as energy consumption costs, cloud resource costs, and carbon emission costs), thereby achieving multi-objective-oriented test task scheduling.

[0029] Figure 1 This is a flowchart illustrating a task scheduling method for an artificial intelligence algorithm testing task according to an embodiment of the present disclosure.

[0030] like Figure 1 As shown, the task scheduling method for testing artificial intelligence algorithms includes steps S101 to S105.

[0031] In step S101, task feature information for each of multiple reference test tasks is obtained. These multiple reference test tasks include at least one task type. The task feature information includes first reference information associated with the task metadata and test dataset of the reference test tasks, second reference information associated with the resource configuration data of the reference test tasks, and third reference information associated with the energy consumption data and multi-dimensional cost data of the reference test tasks.

[0032] In the example, task feature information of reference test tasks can be collected from historical test data. The reference test tasks mentioned herein can refer to historical test tasks that have already been completed. Multiple reference test tasks can include one or more task types. Task types can include, for example, knowledge question answering tests, image classification tests, mathematical reasoning tests, etc. Taking these three task types as examples, the same number of reference test tasks can be obtained for each task type, or a different number of reference test tasks can be obtained for each task type; the embodiments of this disclosure do not impose any limitations on this.

[0033] In the example, task metadata can be used to define the inherent attributes of the test task, such as algorithm model type (e.g., BERT, ResNet, GPT), task scenario (e.g., knowledge answering, image classification, mathematical reasoning), and the number of model parameters. The test dataset can have a corresponding name, size (e.g., number of records or size), and data type (e.g., text or image). Therefore, the first reference information associated with the task metadata and the test dataset can include information from the aspects described above.

[0034] In the example, resource configuration data may refer to the hardware resource specifications used to perform the test task, such as computing resource specifications (e.g., GPU type, memory size, single machine or cluster). Therefore, the second reference information associated with the resource configuration data may include information from the aspects described above.

[0035] In the example, energy consumption data can be obtained by multiplying the power consumed in performing the test task, the time spent performing the test task, and the data center's Power Usage Effectiveness (PUE). The product of power consumed and time spent represents hardware resource energy consumption data, which may depend on the inherent properties of the test task itself and the specific hardware resources used to perform it. PUE can reflect the additional energy consumption of the data center. The product of these three factors yields the overall energy consumption data. Multi-dimensional cost data can include at least two dimensions: economic cost data and environmental cost data. Economic cost data can include electricity cost data or cloud service cost data, while environmental cost data can include carbon emission cost data. Electricity cost data or cloud service cost data can be obtained by multiplying energy consumption data by electricity price or cloud price, and carbon emission cost data can be obtained by multiplying energy consumption data by carbon intensity. Therefore, the third reference information associated with the energy consumption data and multi-dimensional cost data can include information from the aspects described above.

[0036] In step S102, based on at least one task feature information corresponding to each task type, a first prediction model and a second prediction model are constructed for that task type. The first prediction model is constructed based on reference information related to energy consumption data from the first, second, and third reference information, and is used to predict the energy consumption data of the current test task with the corresponding task type. The second prediction model is constructed based on reference information related to multi-dimensional cost data from the first, second, and third reference information, and is used to predict the multi-dimensional cost data of the current test task.

[0037] In the example, the first prediction model, also known as the energy consumption model, is based on historical test data and predicts the energy consumption data of a specific type of test task and specific hardware resource specifications. The first prediction model can be expressed as E = f(x) × PUE, where E represents the predicted energy consumption data, f(x) represents the hardware resource energy consumption data of the test task, and PUE represents the data center energy efficiency coefficient. That is, the first prediction model can establish a mapping relationship between test tasks, hardware resource specifications, and energy consumption data based on historical test data, thereby predicting the corresponding energy consumption data given a specific test task and hardware resource specifications.

[0038] In this example, the second prediction model, also known as the cost model, is capable of modeling based on historical test data. When a specific type of test task and specific hardware resource specifications are received, it predicts the multi-dimensional cost data for that task. As mentioned earlier, the multi-dimensional cost data can include at least two dimensions: economic cost data and environmental cost data. Economic cost data can include electricity cost data or cloud service cost data, while environmental cost data can include carbon emission cost data. The second prediction model can be expressed as C = g(x), where C represents the predicted multi-dimensional cost data, and g(x) can be obtained based on the expression E × electricity price or cloud price in the case of electricity cost data or cloud service cost data, and based on the expression E × carbon intensity in the case of carbon emission cost data, where E represents the predicted energy consumption data. In other words, the second prediction model can establish a mapping relationship between test tasks and hardware resource specifications and multi-dimensional cost data based on historical test data, thereby predicting the corresponding multi-dimensional cost data given a specific test task and hardware resource specifications.

[0039] In the example, the constructed first and second prediction models can be continuously updated and iterated to obtain more accurate predictions, which can be achieved through online incremental calibration.

[0040] In step S103, constraint information for executing the current test task and degradable information for satisfying the constraints are obtained. The constraint information includes first sub-constraint information associated with energy consumption data and multi-dimensional cost data, and second sub-constraint information associated with the test effect data of the current test task.

[0041] In the example, constraint information can be used to indicate the overall constraints for executing the current test task, and this constraint can be multi-dimensional. The first sub-constraint information can be used to characterize budget constraints, while the second sub-constraint information can be used to characterize test quality constraints. Test performance data corresponding to test quality constraints can include, for example, test coverage, regression test completion rate, and test on-time completion rate.

[0042] In the example, degradability information can be used to indicate the execution standards that are allowed to be reduced to accommodate the constraints specified in the constraint information. For example, degradability information can specify which execution standards are allowed to be reduced, such as the test precision mode (allowing a reduction from high precision to low precision), the test review ratio (allowing a reduction from full review to sampling review), etc. Similarly, degradability information can also specify which execution standards are not allowed to be reduced, i.e., a non-degradable whitelist. For example, this non-degradable whitelist can specify that test tasks for critical test scenarios are not degradable, test tasks for critical versions are not degradable, and the deadlines for test task completion are not degradable.

[0043] In step S104, based on the first prediction model, the second prediction model, and the constraint information, it is determined whether there exists a first candidate scheduling result for executing the current test task that simultaneously satisfies the first sub-constraint information and the second sub-constraint information.

[0044] In the example, step S104 can be performed by finding feasible solutions, and the first candidate scheduling result can be one of the found feasible solutions. For example, to execute the current test task, it is necessary to determine the batches into which the current test task is divided and the test priority order of the test dataset. Since the batches and priority order themselves are variable, different budget results and test performance results may result from applying different batches and / or different priority orders. Since the task type of the current test task can be known, the budget result can be predicted by the first prediction model and the second prediction model corresponding to the task type, for example, it can be predicted according to different hardware resource specifications, and the budget result needs to meet the budget constraints of the first sub-constraint information, so possible batches and / or priority orders can be solved. In this case, since the test quality requirements of the second sub-constraint information also need to be met, a portion of the possible batches and / or priority orders that do not meet the requirements can be excluded. The batch may affect whether the test task can be completed or how much can be executed, and the priority order may affect whether the key test scenarios are covered by the test. If there are still possible batches and / or priority orders remaining, they can be included in the first candidate scheduling result. In other words, since the current test task has the possibility of being executed in various different ways, determining the first candidate scheduling result means selecting at least one solution from these different ways that can simultaneously satisfy the first sub-constraint information and the second sub-constraint information, i.e., the feasible solution.

[0045] In step S105, in response to determining that there is no first candidate scheduling result, a second candidate scheduling result for executing the current test task is determined based on the degradability information.

[0046] In the example, if no first candidate scheduling result exists, it means that a feasible solution cannot be found according to the strictest criteria of the first and second sub-constraint information. In this case, a feasible solution, i.e., the second candidate scheduling result, is searched again using an appropriate adjustable strategy. As mentioned earlier, degradability information can be used to indicate the execution criteria that can be reduced to adapt to the constraints specified in the constraint information. In other words, degradability information can be used in the aforementioned adjustable strategy. However, although there are many ways to execute the current test task, due to the limitations of degradability information, adjustable strategies can only be adopted within the range indicated by the degradability information to find the second candidate scheduling result.

[0047] Therefore, in the task scheduling method for testing artificial intelligence algorithms according to the embodiments of this disclosure, a task profile model constructed based on historical test data can be used to achieve multi-objective-oriented test task scheduling by means of adjustable dynamic strategy linkage, while ensuring test results and meeting given multi-dimensional cost requirements, thereby achieving synergistic optimization of test efficiency, cost control and energy conservation and emission reduction.

[0048] The following describes in further detail various aspects of a task scheduling method for testing artificial intelligence algorithms according to embodiments of the present disclosure.

[0049] In some embodiments, each of the first candidate scheduling result and the second candidate scheduling result may include a first sub-scheduling result associated with the scheduling strategy of the current test task, a second sub-scheduling result associated with the resource configuration data of the current test task, and a third sub-scheduling result associated with the energy consumption data and multi-dimensional cost data of the current test task.

[0050] In the example, the scheduling strategy can refer to the scheduling method for executing the current test task. Resource configuration data can refer to the hardware resource specifications used to execute the current test task. Energy consumption data and multi-dimensional cost data can be predicted based on the first prediction model and the second prediction model, respectively.

[0051] By characterizing the scheduling results in multiple dimensions, not only can the test quality requirements be transformed into practically executable solutions, but energy consumption and cost budgets can also be transformed from implicit constraints into explicit indicators to participate in the overall task scheduling decision-making. This leads to task scheduling results no longer being driven by pure efficiency as used in traditional methods, but rather by solutions optimized under multi-objective constraints.

[0052] In some embodiments, the first sub-scheduling result may include at least one of the following: the batch into which the current test task is divided, the test priority order of the test dataset, the test precision mode, the test review ratio, the test period, and the test concurrency. The second sub-scheduling result may include the computing resource specifications used for the current test task.

[0053] In the example, since the scheduling strategy can be dynamic, the first sub-scheduling result can have dynamically adjustable attributes. The batches into which the current test task is divided can be related to the size of the test dataset and the amount of test data processed at one time. For example, if there are a total of 100,000 test data points and 1,000 test data points are processed at one time, it means that 100 batches are needed to complete the current test task. The test priority order of the test datasets can indicate the order in which the various test datasets are tested. For example, in a knowledge question answering test task, datasets related to common sense reasoning can be tested first, followed by datasets related to professional domains. The test precision mode can indicate the computational precision level, such as high-precision FP32 and low-precision FP16. The test verification ratio can indicate the sampling ratio used for result verification, such as 100% full verification, 30% sampling verification, etc. The test period can indicate the time period of the day during which the current test task is executed. The test concurrency can indicate the amount of computing resources that can execute the current test task in parallel, such as single-card serial or multi-card parallel.

[0054] In the example, since computing resource specifications can also be dynamic, the second sub-scheduling result can also have dynamically adjustable attributes. Computing resource specifications can include processor type, memory capacity, number of compute nodes, etc.

[0055] By dividing a portion of the multi-dimensional scheduling results into more granular adjustable policy parameters and separating them from equally adjustable resource configurations, a decoupling design between test execution strategies and hardware resources can be achieved. This decoupling allows for rapid response to changes in energy consumption or cost constraints without altering the underlying hardware investment; adjustments can be made to execution parameters such as batch size, precision mode, and test duration. This enables flexible cost control and energy-saving optimization while ensuring test quality.

[0056] In some embodiments, the degradability information may indicate the permissible range within which the first sub-scheduling result and the second sub-scheduling result can be adjusted, provided that the first sub-constraint information and the second sub-constraint information are satisfied.

[0057] In the example, as mentioned earlier, since the first and second sub-scheduling results can have dynamic adjustment attributes, it means that the first and second sub-scheduling results can be adjusted. However, under the dual constraints of the budget constraint indicated by the first sub-constraint information and the test quality constraint indicated by the second sub-constraint information, the first and second sub-scheduling results do not have an unrestricted adjustable range, but rather a defined adjustable range indicated by the degradability information.

[0058] Therefore, by using degradable information, the adjustable boundaries of the first and second sub-scheduling results are clearly defined, achieving a controllable balance between constraint satisfaction and strategy adjustment. This helps to maximize utilization efficiency and achieve the optimal trade-off between energy saving and quality while ensuring hard constraints.

[0059] Figure 2 This is a schematic diagram illustrating the first candidate scheduling result and the second candidate scheduling result according to an embodiment of the present disclosure.

[0060] like Figure 2 As shown, the first or second candidate scheduling result includes a first sub-scheduling result 201, a second sub-scheduling result 202, and a third sub-scheduling result 203. The first sub-scheduling result 201 is associated with the scheduling strategy of the current test task and may include at least one of the following: the batch into which the current test task is divided, the test priority order of the test dataset, the test precision mode, the test review ratio, the test period, and the test concurrency. The second sub-scheduling result 202 is associated with the resource configuration data of the current test task and may include the computing resource specifications used for the current test task. The third sub-scheduling result 203 is associated with the energy consumption data and multi-dimensional cost data of the current test task. The first and second sub-scheduling results 201 and 202 are dynamically adjustable. Accordingly, the degradable range of the first and second sub-scheduling results 201 and 202 can be indicated by degradable information, which is subject to dual constraints from the first and second sub-constraint information.

[0061] In some embodiments, during the execution of the current test task based on the first candidate scheduling result or the second candidate scheduling result, the processes of actively optimizing the scheduling result and passively correcting the scheduling result can also be performed.

[0062] Figure 3 This is a flowchart illustrating a task scheduling method for an artificial intelligence algorithm testing task according to another embodiment of the present disclosure.

[0063] like Figure 3 As shown, during the execution of the current test task based on the first candidate scheduling result or the second candidate scheduling result, the process of actively optimizing the scheduling result, including steps S311 and S312, and the process of passively correcting the scheduling result, including steps S321 and S322, can be executed.

[0064] In step S311, it is determined whether either the energy consumption data monitoring data or the multi-dimensional cost data monitoring data is lower than the corresponding predicted data and reaches a first predetermined threshold.

[0065] In the example, the multi-dimensional cost data is the product of energy consumption data and unit energy cost parameters. Unit energy cost parameters can include electricity prices, cloud prices, or carbon intensity, such as the price per kilowatt-hour, the cloud service price corresponding to a unit of energy consumption, and the carbon emissions per kilowatt-hour. Therefore, when the unit energy cost parameter fluctuates to the point that the deviation between the monitored data and the predicted data reaches a certain level (such as a significant reduction in electricity prices), it presents an opportunity to optimize the scheduling results.

[0066] In the example, monitoring data can be correlated with the real-time execution of either the first or second candidate scheduling result, while predicted data can be correlated with the constraints used to determine the first or second candidate scheduling result. Therefore, determining whether the monitoring data falls below the predicted data to reach a predetermined threshold aims to judge through real-time execution whether there is an opportunity to optimize the scheduling result in a more lenient direction, i.e., whether there is room for optimization to improve the scale or quality of test execution. The first predetermined threshold can be set according to the actual situation to characterize the timing for optimizing the scheduling result.

[0067] In step S312, in response to the determination that either the monitoring data of energy consumption data or the monitoring data of multi-dimensional cost data is lower than the corresponding predicted data to reach a first predetermined threshold, the first candidate scheduling result or the second candidate scheduling result is optimized based on the allowable range that can be adjusted in the first sub-scheduling result and the second sub-scheduling result.

[0068] In the example, if it is determined that there is an opportunity to optimize the scheduling result, one can start by adjusting the allowable range between the first and second sub-scheduling results to optimize either the first or second candidate scheduling result. For example, one could employ low-cost / low-PUE time period migration, energy-efficient resource selection, precision grading (such as low precision, sampling verification, etc.), concurrency adaptation, queuing and preemption guarantee strategies, etc.

[0069] In step S321, it is determined whether either the energy consumption data monitoring data or the multi-dimensional cost data monitoring data exceeds the corresponding predicted data and reaches a second predetermined threshold.

[0070] In the example, compared to the optimization scenario where the monitored data falls below the predicted data to a predetermined threshold, the scenario where the monitored data exceeds the predicted data to a predetermined threshold corresponds to a correction scenario. This means that the real-time execution may be exceeding budget constraints or the budget is being consumed too quickly, requiring correction to tighten the scale or quality of test execution. The second predetermined threshold can be set according to the actual situation to characterize when to correct the scheduling results.

[0071] In step S322, in response to the determination that either the monitoring data of energy consumption data or the monitoring data of multi-dimensional cost data is higher than the corresponding predicted data to reach a second predetermined threshold, the first candidate scheduling result or the second candidate scheduling result is corrected based on the allowable range that can be adjusted in the first sub-scheduling result and the second sub-scheduling result.

[0072] In the example, if it is determined that the scheduling result needs to be corrected, one can start by considering the allowable range of adjustment between the first and second sub-scheduling results, and then correct either the first or second candidate scheduling result. For example, dynamic reordering, reducing precision / concurrency, migrating or pausing non-critical tasks, and suppressing high energy consumption under low load by turning nodes on / off or into hibernation can be employed.

[0073] By proactively optimizing scenario settings, the scale of test execution can be dynamically expanded or test quality improved, maximizing test benefits within budget constraints. Conversely, by passively adjusting scenario settings, the scale of test execution can be reduced or test quality lowered in a timely manner, ensuring that budget constraints are not exceeded. The combination of these two approaches forms a closed-loop control system, enabling test scheduling to capitalize on opportunities for efficiency improvements while mitigating risks to ensure compliance with constraints, thus contributing to a dynamic balance between energy consumption, cost, and quality.

[0074] In some embodiments, the first candidate scheduling result and the second candidate scheduling result may be obtained through iterative optimization based on heuristic or hierarchical strategies.

[0075] In the example, heuristic strategies can include quickly finding feasible solutions based on empirical rules without exhaustively exploring all possibilities, such as prioritizing tasks during periods of lowest electricity prices, ensuring all whitelisted tasks are fully and accurately processed, and directly sampling and verifying non-critical tasks. Therefore, these empirical rules can be used to quickly generate solutions that meet the constraints.

[0076] In the example, a hierarchical strategy could include prioritizing optimization layer by layer, with each layer addressing a specific type of problem. For instance, the first layer might prioritize hard constraints, the second layer might optimize quality, and the third layer might optimize cost, and so on. Therefore, each layer optimizes based on the conditions of the previous layer, converging layer by layer to the final solution.

[0077] Therefore, by employing heuristic or hierarchical strategies for iterative optimization, a feasible scheduling scheme that satisfies multiple constraints can be quickly converged without exhaustively exploring all possible configuration combinations. The combination of these two approaches ensures that test plan generation is both computationally efficient and guarantees the controllability of constraint satisfaction, thus adapting to scenarios characterized by large-scale algorithm testing tasks, numerous constraints, and high timeliness.

[0078] In some embodiments, the first and second prediction models may be constructed based on linear regression modeling, decision tree modeling, or piecewise linear modeling.

[0079] In the examples, linear regression modeling is applicable to scenarios where task features are approximately linearly correlated with energy consumption or cost, providing a stable prediction benchmark with minimal computational overhead. Decision tree modeling can segment and process high-dimensional discrete features through hierarchical rules, effectively capturing the differentiated impact of discontinuous variables such as model type and resource allocation on energy consumption or cost. Piecewise linear modeling can address situations where the same feature exhibits heterogeneous patterns across different value ranges, achieving local linear fitting through interval division and improving the prediction accuracy of nonlinear relationships.

[0080] By employing machine learning methods such as linear regression, decision tree modeling, or piecewise linear modeling to construct energy consumption and cost prediction models, it is possible to adapt to data feature relationships of varying complexity in algorithm testing tasks while ensuring predictive interpretability. These different modeling methods each have their applicable scenarios and can be flexibly selected or combined based on historical data distribution characteristics and prediction accuracy requirements. This ensures the model is lightweight and deployable while providing reliable predictive input for subsequent constraint optimization.

[0081] In some embodiments, the task scheduling method may also include steps associated with energy-saving effect assessment and report output. For example, an audit report may be output, such as a comparison of actual energy consumption / cost / carbon emissions with the budget, coverage and regression indicators, strategy triggering and degradation statistics, and a comparison of energy-saving benefits and quality impacts with a baseline.

[0082] According to embodiments of this disclosure, a task scheduling apparatus for testing artificial intelligence algorithms is also provided.

[0083] Figure 4 This is a block diagram illustrating a task scheduling apparatus for testing artificial intelligence algorithms according to an embodiment of the present disclosure.

[0084] like Figure 4 As shown, in some embodiments, the task scheduling device 400 for testing artificial intelligence algorithms includes a modeling information acquisition module 401, a prediction model construction module 402, a constraint information acquisition module 403, a first scheduling determination module 404, and a second scheduling determination module 405.

[0085] The modeling information acquisition module 401 is configured to acquire task feature information of each of a plurality of reference test tasks, wherein the plurality of reference test tasks include at least one task type, and the task feature information includes first reference information associated with the task metadata and test dataset of the reference test task, second reference information associated with the resource configuration data of the reference test task, and third reference information associated with the energy consumption data and multi-dimensional cost data of the reference test task.

[0086] The prediction model building module 402 is configured to build a first prediction model and a second prediction model for each task type based on at least one task feature information corresponding to that task type. The first prediction model is built based on reference information related to energy consumption data from the first reference information, the second reference information, and the third reference information, and is used to predict the energy consumption data of the current test task with the corresponding task type. The second prediction model is built based on reference information related to multi-dimensional cost data from the first reference information, the second reference information, and the third reference information, and is used to predict the multi-dimensional cost data of the current test task.

[0087] The constraint information acquisition module 403 is configured to acquire constraint information for executing the current test task, and degradable information for satisfying the constraint information. The constraint information includes first sub-constraint information associated with energy consumption data and multi-dimensional cost data, and second sub-constraint information associated with the test effect data of the current test task.

[0088] The first scheduling determination module 404 is configured to determine, based on the first prediction model, the second prediction model, and constraint information, whether there exists a first candidate scheduling result for executing the current test task that simultaneously satisfies the first sub-constraint information and the second sub-constraint information.

[0089] The second scheduling determination module 405 is configured to determine a second candidate scheduling result for executing the current test task based on degradability information in response to determining that there is no first candidate scheduling result.

[0090] The operations of the aforementioned modeling information acquisition module 401, prediction model construction module 402, constraint information acquisition module 403, first scheduling determination module 404, and second scheduling determination module 405 can be combined. Figure 1 The operations of steps S101 to S105 are the same, so the details of each aspect will not be repeated here.

[0091] In some embodiments, the task scheduling apparatus 400 may further include an optimization determination module 406 and an optimization execution module 407. The optimization determination module 406 is configured to determine, during the execution of the current test task based on a first candidate scheduling result or a second candidate scheduling result, whether either the energy consumption data monitoring data or the multi-dimensional cost data monitoring data falls below a corresponding predicted data to reach a predetermined threshold. The optimization execution module 407 is configured to, in response to determining that either the energy consumption data monitoring data or the multi-dimensional cost data monitoring data falls below a corresponding predicted data to reach a predetermined threshold, optimize the first candidate scheduling result or the second candidate scheduling result based on an adjustable allowable range in the first and second sub-scheduling results.

[0092] In some embodiments, the task scheduling apparatus 400 may further include a correction determination module 408 and a correction execution module 409. The correction determination module 408 is configured to determine, during the execution of the current test task based on a first candidate scheduling result or a second candidate scheduling result, whether either the energy consumption data monitoring data or the multi-dimensional cost data monitoring data exceeds a corresponding predicted data by a predetermined threshold. The correction execution module 409 is configured to, in response to determining that either the energy consumption data monitoring data or the multi-dimensional cost data monitoring data exceeds a corresponding predicted data by a predetermined threshold, correct the first candidate scheduling result or the second candidate scheduling result based on an adjustable allowable range in the first and second sub-scheduling results.

[0093] According to one aspect of this disclosure, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0094] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0095] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0096] Figure 5 An example computer device 500 is shown in which any of the embodiments described herein may be implemented. The computer device 500 may be used to implement one or more components of the systems and methods described above. The computer device 500 may include a bus 502 or other communication mechanism for communicating information, and one or more processors 504 coupled to the bus 502 for processing information. The processor 504 may be, for example, one or more general-purpose microprocessors.

[0097] Computer device 500 may also include main memory 506, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 502, for storing information and instructions to be executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 504. Such instructions, when stored in a storage medium accessible to processor 504, can make computer device 500 a special-purpose machine customized to perform the operations specified in the instructions. Main memory 506 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common media formats may include, for example, floppy disks, collapsible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs (read-only optical disc drives), any other optical data storage media, any physical media with a perforated arrangement, RAM (random access memory), DRAM (dynamic random access memory), PROM (programmable read-only memory) and EPROM (erasable programmable read-only memory), FLASH-EPROM (fast erase programmable read-only memory), NVRAM (non-volatile random access memory), any other memory chips or tape cartridges, or network versions of the above.

[0098] Computer device 500 may implement the techniques described herein using custom hardwired logic, one or more ASICs (Application-Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays), firmware, and / or program logic, which, when combined with computer device 500, enable computer device 500 to become a special-purpose machine or to be programmed therein. According to one embodiment, the techniques described herein are executed by computer device 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 508. Executing the sequence of instructions contained in main memory 506 causes processor 504 to perform the processing steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in main memory 506. When these instructions are executed by processor 504, they may perform the steps shown in the corresponding figures and as described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0099] Computer device 500 also includes a network interface 510 coupled to bus 502. Network interface 510 can provide bidirectional data communication coupled to one or more network links connected to one or more networks. As another example, network interface 510 can be a local area network (LAN) card to provide data communication connectivity with a compatible LAN (or a WAN component communicating with a WAN (wide area network)). Wireless links can also be implemented.

[0100] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0101] Each process, method, and algorithm described in the preceding sections can be embodied in a code module executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by them. These processes and algorithms can be implemented, in part or in whole, in a specific application circuit.

[0102] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, can be embodied in the form of a software product. This software product can be stored in a storage medium and includes instructions to cause a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods described in the embodiments of this application. The storage medium may include a flash drive, a portable hard drive, ROM, RAM, a magnetic disk, an optical disk, another medium suitable for storing program code, or any combination thereof.

[0103] The embodiments disclosed herein can be implemented via a cloud platform, server, or group of servers that interact with a client. The client can be a terminal device or a client registered by a user on the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can install platform applications.

[0104] The various features and processes described above can be used independently or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in a non-specifically disclosed order, or multiple blocks or states may be combined in a single block or state. Exemplary blocks or states may be executed serially, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed exemplary embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed exemplary embodiments.

[0105] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. An algorithm may consist of program code or instructions stored in memory (such as the non-transitory computer-readable storage medium described above). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not be explicitly programmed into the computer to perform the function, but may learn from training data to obtain a predictive model for performing that function.

[0106] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute the engine of a processor implementation whose operation is to perform one or more of the operations or functions described herein.

[0107] Similarly, the methods described herein can be implemented at least partially by a processor, where a specific processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also run in a “cloud computing” environment or as “Software as a Service” (SaaS) to support the execution of the relevant operations. For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), which can be accessed via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0108] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0109] In this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are described and illustrated as independent operations, one or more individual operations may be performed concurrently, and these operations are not required to be performed in the order shown. Structures and functionalities presented as independent components in the example configuration may be implemented as combined structures or components. Similarly, structures and functionalities presented as individual components may be implemented as independent components. These and other variations, modifications, additions, and improvements are all within the scope of this document.

[0110] As used herein, “or” is inclusive rather than exclusive unless explicitly stated or indicated by context. Furthermore, “and” is both common and individual unless explicitly stated or indicated by context. Moreover, multiple instances may be provided for the resources, operations, or structures described herein as a single example. Furthermore, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and specific operations are illustrated within the context of a particular illustrative configuration. The allocation of other functionalities is conceivable and may fall within the scope of various embodiments of this disclosure. Generally, structures and functionalities presented as independent resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functionalities presented as individual resources may be implemented as independent resources. These and other variations, modifications, additions, and improvements are all within the scope of embodiments of this disclosure. Therefore, this specification and accompanying drawings should be viewed in an illustrative rather than restrictive sense.

[0111] The terms “comprising” or “including” are used to indicate the presence of a subsequently stated feature, but do not preclude the addition of other features. Conditional language, in particular, such as “may,” “can,” or “may,” unless specifically stated or otherwise understood in the context of use, is generally intended to express that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Therefore, such conditional language generally does not imply that a feature, element, and / or step is necessary in any way for one or more embodiments, or that one or more embodiments must include logic that, with or without user input or prompting, determines whether such features, elements, and / or steps are included in any particular embodiment, or whether they are to be performed in any particular embodiment.

Claims

1. A task scheduling method for testing artificial intelligence algorithms, characterized in that, The method includes: Obtain task feature information for each of multiple reference test tasks, wherein the multiple reference test tasks include at least one task type, and the task feature information includes first reference information associated with the task metadata and test dataset of the reference test task, second reference information associated with the resource configuration data of the reference test task, and third reference information associated with the energy consumption data and multi-dimensional cost data of the reference test task. Based on at least one task feature information corresponding to each task type, a first prediction model and a second prediction model are constructed for that task type. The first prediction model is constructed based on reference information related to the energy consumption data from the first reference information, the second reference information, and the third reference information, and is used to predict the energy consumption data of the current test task with the corresponding task type. The second prediction model is constructed based on reference information related to the multi-dimensional cost data from the first reference information, the second reference information, and the third reference information, and is used to predict the multi-dimensional cost data of the current test task. Obtain constraint information for executing the current test task, and degradable information for satisfying the constraint information, wherein the constraint information includes first sub-constraint information associated with the energy consumption data and the multi-dimensional cost data, and second sub-constraint information associated with the test effect data of the current test task; Based on the first prediction model, the second prediction model, and the constraint information, determine whether there exists a first candidate scheduling result for executing the current test task that simultaneously satisfies the first sub-constraint information and the second sub-constraint information; and In response to the determination that the first candidate scheduling result does not exist, a second candidate scheduling result for executing the current test task is determined based on the degradability information.

2. The method according to claim 1, characterized in that, Each of the first candidate scheduling result and the second candidate scheduling result includes a first sub-scheduling result associated with the scheduling strategy of the current test task, a second sub-scheduling result associated with the resource configuration data of the current test task, and a third sub-scheduling result associated with the energy consumption data and the multi-dimensional cost data of the current test task.

3. The method according to claim 2, characterized in that, The first sub-scheduling result includes at least one of the following: the batch into which the current test task is divided, the test priority order of the test dataset, the test precision mode, the test review ratio, the test period, and the test concurrency. The second sub-scheduling result includes the computing resource specifications used for the current test task.

4. The method according to claim 2, characterized in that, The degradability information indicates the permissible range within which the first sub-scheduling result and the second sub-scheduling result can be adjusted, provided that the first sub-constraint information and the second sub-constraint information are satisfied.

5. The method according to claim 4, characterized in that, in, The method further includes: During the execution of the current test task based on the first candidate scheduling result or the second candidate scheduling result, it is determined whether either the energy consumption data monitoring data or the multi-dimensional cost data monitoring data falls below the corresponding predicted data to reach a first predetermined threshold; and In response to determining that either the monitoring data of the energy consumption data or the monitoring data of the multi-dimensional cost data is lower than the corresponding predicted data to reach a first predetermined threshold, the first candidate scheduling result or the second candidate scheduling result is optimized based on the allowable range that can be adjusted in the first sub-scheduling result and the second sub-scheduling result.

6. The method according to claim 4, characterized in that, The method further includes: During the execution of the current test task based on the first candidate scheduling result or the second candidate scheduling result, it is determined whether either the energy consumption data monitoring data or the multi-dimensional cost data monitoring data exceeds the corresponding predicted data to reach a second predetermined threshold; and In response to determining that either the monitoring data of the energy consumption data or the monitoring data of the multi-dimensional cost data is higher than the corresponding predicted data to reach a second predetermined threshold, the first candidate scheduling result or the second candidate scheduling result is corrected based on the allowable range that can be adjusted in the first sub-scheduling result and the second sub-scheduling result.

7. The method according to any one of claims 1 to 6, characterized in that, The first candidate scheduling result and the second candidate scheduling result are obtained through iterative optimization based on heuristic or hierarchical strategies.

8. The method according to any one of claims 1 to 6, characterized in that, The first prediction model and the second prediction model are constructed based on linear regression modeling, decision tree modeling, or piecewise linear modeling.

9. A task scheduling device for testing artificial intelligence algorithms, characterized in that, The device includes: The modeling information acquisition module is configured to acquire task feature information of each of multiple reference test tasks, wherein the multiple reference test tasks include at least one task type, and the task feature information includes first reference information associated with the task metadata and test dataset of the reference test task, second reference information associated with the resource configuration data of the reference test task, and third reference information associated with the energy consumption data and multi-dimensional cost data of the reference test task. The prediction model building module is configured to build a first prediction model and a second prediction model for each task type based on at least one task feature information corresponding to that task type. The first prediction model is built based on reference information related to the energy consumption data from the first reference information, the second reference information, and the third reference information, and is used to predict the energy consumption data of the current test task with the corresponding task type. The second prediction model is built based on reference information related to the multi-dimensional cost data from the first reference information, the second reference information, and the third reference information, and is used to predict the multi-dimensional cost data of the current test task. The constraint information acquisition module is configured to acquire constraint information for executing the current test task, and degradable information for satisfying the constraint information, wherein the constraint information includes first sub-constraint information associated with the energy consumption data and the multi-dimensional cost data, and second sub-constraint information associated with the test effect data of the current test task; The first scheduling determination module is configured to determine, based on the first prediction model, the second prediction model, and the constraint information, whether there exists a first candidate scheduling result for executing the current test task that simultaneously satisfies the first sub-constraint information and the second sub-constraint information; and The second scheduling determination module is configured to determine a second candidate scheduling result for executing the current test task based on the degradability information in response to determining that the first candidate scheduling result does not exist.

10. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-8.

Images