Resource scheduling method, device, equipment and storage medium based on intelligent computing cluster

Abstract

The embodiments of the present application provide a resource scheduling method, device, equipment and storage medium based on intelligent computing clusters. The method includes: benchmarking multiple intelligent computing clusters to obtain the energy consumption proportional factor and duration proportional factor of each first cluster relative to the benchmark cluster; predicting the energy consumption and duration of each task to be processed by the benchmark cluster to obtain the benchmark energy consumption and benchmark duration; converting the energy consumption and duration of each first cluster to obtain the corresponding first energy consumption and first duration; inputting the benchmark energy consumption and benchmark duration corresponding to the benchmark cluster, and the first energy consumption and first duration corresponding to each first cluster into a preset target model to obtain a model to be solved; solving based on the model to be solved, determining the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters, and allocating multiple tasks to be processed to multiple intelligent computing clusters. In this way, computing power resources can be reasonably utilized, which is conducive to the sustainable development of the environment.

Description

Technical Field

[0001] The present application relates to the field of resource scheduling, and in particular to a resource scheduling method, device, equipment and storage medium based on an intelligent computing cluster. Background Art

[0002] An intelligent computing cluster is a computing resource and architecture that combines high-performance computing capabilities with intelligent technologies (such as artificial intelligence, big data analysis, etc.). It aims to improve the utilization of computing resources, reduce energy consumption, and accelerate the execution of complex computing tasks through intelligent management and optimization strategies.

[0003] However, the physical facilities or operating environments (i.e., data centers) used to host and run these intelligent computing clusters are energy-intensive, accounting for a significant proportion of the total electricity consumption in society. As the demand for computing power for artificial intelligence increases, the energy consumption problem is becoming increasingly prominent. Therefore, it is necessary to reduce the energy consumption and carbon emissions of intelligent computing clusters in order to provide powerful computing power for artificial intelligence technology while reducing the negative impact on the environment.

[0004] In related technologies, when tasks are assigned to clusters for processing, the order in which each intelligent computing cluster processes tasks is usually determined by the order in which the tasks are submitted. After the tasks are submitted to the scheduling system, they will be recorded and queued. The scheduler will allocate them to appropriate computing resources or clusters based on the order in which the tasks enter the queue. However, this allocation method makes it impossible to make reasonable use of computing resources, which is not conducive to the sustainable development of the environment. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a resource scheduling method, device, equipment and storage medium based on an intelligent computing cluster, which can make rational use of computing resources and be conducive to the sustainable development of the environment.

[0006] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a resource scheduling method based on an intelligent computing cluster, the method comprising:

[0007] Benchmark multiple intelligent computing clusters corresponding to multiple regions using a benchmark test program to obtain an energy consumption proportional factor and a duration proportional factor of each first cluster relative to the benchmark cluster;

[0008] Acquire multiple tasks to be processed and multiple historical processing tasks of the multiple intelligent computing clusters, and predict the energy consumption and duration of each task to be processed by the benchmark cluster through metadata differences between the multiple historical processing tasks and the multiple tasks to be processed, so as to obtain benchmark energy consumption and benchmark duration;

[0009] According to the reference energy consumption and the energy consumption proportional factor, convert the energy consumption of each first cluster to obtain a corresponding first energy consumption; according to the reference duration and the duration proportional factor, convert the duration of each first cluster to obtain a corresponding first duration;

[0010] Input the benchmark energy consumption and the benchmark duration corresponding to the benchmark cluster, and the first energy consumption and the first duration corresponding to each first cluster into a preset target model to obtain a model to be solved, wherein the model to be solved includes a comprehensive energy consumption index under the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters;

[0011] The comprehensive energy consumption index is minimized based on the model to be solved, an allocation relationship between the multiple tasks to be processed and the multiple intelligent computing clusters is determined, and the multiple tasks to be processed are allocated to the multiple intelligent computing clusters according to the allocation relationship.

[0012] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a resource scheduling device based on an intelligent computing cluster, the device comprising:

[0013] A testing module, used to perform a benchmark test on multiple intelligent computing clusters corresponding to multiple regions through a benchmark test program to obtain an energy consumption proportional factor and a duration proportional factor of each first cluster relative to the benchmark cluster;

[0014] A prediction module is used to obtain multiple tasks to be processed and multiple historical processing tasks of the multiple intelligent computing clusters, and predict the energy consumption and duration of each task to be processed by the benchmark cluster through the metadata differences between the multiple historical processing tasks and the multiple tasks to be processed, so as to obtain a benchmark energy consumption and a benchmark duration;

[0015] a conversion module, configured to convert the energy consumption of each of the first clusters according to the reference energy consumption and the energy consumption proportional factor to obtain a corresponding first energy consumption, and to convert the duration of each of the first clusters according to the reference duration and the duration proportional factor to obtain a corresponding first duration;

[0016] An input module, used for inputting the benchmark energy consumption and the benchmark duration corresponding to the benchmark cluster, and the first energy consumption and the first duration corresponding to each first cluster into a preset target model to obtain a model to be solved, wherein the model to be solved includes a comprehensive energy consumption index under the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters;

[0017] A solution module is used to perform a minimization solution of the comprehensive energy consumption index based on the model to be solved, determine the allocation relationship between the multiple tasks to be processed and the multiple intelligent computing clusters, and allocate the multiple tasks to be processed to the multiple intelligent computing clusters according to the allocation relationship.

[0018] In some embodiments, the input module is further used to:

[0019] According to the benchmark energy consumption and the first energy consumption, input into a preset target model, determining the power consumption function and the carbon emission function of the target model;

[0020] According to the reference duration and the first duration input into the preset target model, a duration occupancy function of the target model is determined;

[0021] Based on the power consumption function, the carbon emission function and the time occupancy function, a model to be solved is obtained.

[0022] In some implementations, the resource scheduling device based on the intelligent computing cluster further includes a construction module for:

[0023] Obtain the initial power consumption index, initial carbon emission index and initial duration occupancy index of each intelligent computing cluster processing the initial assigned initial tasks;

[0024] Obtaining an initial power consumption function according to the product of the sum of the initial power consumption indexes corresponding to the multiple intelligent computing clusters and a preset first weight;

[0025] Obtaining an initial carbon emission function according to the sum of the initial carbon emission indexes corresponding to the multiple intelligent computing clusters and the product of a preset second weight;

[0026] An initial duration occupancy function is obtained according to the sum of the multiple initial duration occupancy indexes corresponding to the multiple intelligent computing clusters and the product of a preset third weight; wherein the first weight, the second weight and the third weight are determined according to a resource emphasis condition;

[0027] Obtain a constraint model constructed by the quantity constraint function and the hardware constraint function set for each intelligent computing cluster, and construct a target model based on the initial power consumption function, the initial carbon emission function, the initial time occupancy function and the constraint model.

[0028] In some embodiments, the building block is further used to:

[0029] Obtain the initial cluster energy consumption value and initial duration occupancy value of each intelligent computing cluster processing the corresponding initial task;

[0030] Obtaining the unit power consumption corresponding to the area where each intelligent computing cluster is located, and determining the initial power consumption index of each intelligent computing cluster when processing the initial task according to the product of the initial cluster energy consumption value and the corresponding unit power consumption;

[0031] Obtaining the unit carbon consumption corresponding to the area where each intelligent computing cluster is located, and determining the initial carbon emission index when each intelligent computing cluster processes the initial task according to the product of the initial cluster energy consumption value and the corresponding unit carbon consumption;

[0032] An initial duration occupancy index when each intelligent computing cluster processes the initial task is determined according to the initial duration occupancy value.

[0033] In some embodiments, the testing module is further used for:

[0034] Benchmarking multiple intelligent computing clusters in a distributed cluster using a benchmark test program to obtain the test energy consumption and test duration of each intelligent computing cluster after running the benchmark test program;

[0035] Determine a benchmark cluster and multiple first clusters from the multiple intelligent computing clusters, and determine an energy consumption proportional factor of each first cluster relative to the benchmark cluster based on the ratio of the test energy consumption of each first cluster to the benchmark cluster, and determine the energy consumption proportional factor of each first cluster relative to the benchmark cluster based on the ratio of the test duration of each first cluster to the benchmark cluster.

[0036] In some embodiments, the prediction module is further used to:

[0037] The pre-trained target model is used to calculate the difference between the multiple first metadata corresponding to each task to be processed and the multiple second metadata corresponding to each historical processing task, so as to obtain a target difference value;

[0038] determining a plurality of reference processing tasks from the plurality of historical processing tasks based on a plurality of target difference values ​​corresponding to the plurality of historical processing tasks;

[0039] The reference energy consumption and reference duration of each reference processing task are obtained, and the energy consumption of the benchmark cluster processing the corresponding tasks to be processed is predicted based on multiple reference energy consumptions to obtain a benchmark energy consumption, and the duration of the benchmark cluster processing the corresponding tasks to be processed is predicted based on multiple reference durations to obtain a benchmark duration.

[0040] In some embodiments, the prediction module is further used to:

[0041] According to the plurality of reference energy consumptions, determining that the intelligent computing cluster that processes the corresponding reference processing tasks is the second cluster;

[0042] Obtaining a reference energy consumption proportional factor and a reference duration proportional factor of each second cluster relative to the reference cluster;

[0043] Based on the reference energy consumption proportional factor, convert the reference energy consumption of the corresponding second cluster to predict the second energy consumption of the reference processing task corresponding to the benchmark cluster, and predict the energy consumption of the benchmark cluster processing the corresponding task to be processed based on the multiple second energy consumptions to obtain the benchmark energy consumption when the benchmark cluster processes the task to be processed;

[0044] Based on the reference duration ratio factor, the reference duration of the corresponding second cluster is converted to predict the second duration of the reference processing task corresponding to the benchmark cluster, and based on multiple second durations, the duration of the benchmark cluster processing the corresponding task to be processed is predicted to obtain the benchmark duration when the benchmark cluster processes the task to be processed.

[0045] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the resource scheduling method based on the intelligent computing cluster described in any one of the embodiments of the first aspect of the present application.

[0046] To achieve the above-mentioned objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the resource scheduling method based on the intelligent computing cluster as described in any one of the embodiments of the first aspect of the present application.

[0047] The embodiment of the present application performs a benchmark test on multiple intelligent computing clusters corresponding to multiple regions through a benchmark test program, obtains an energy consumption proportional factor and a duration proportional factor of each first cluster relative to a benchmark cluster; obtains multiple tasks to be processed and multiple historical processing tasks of multiple intelligent computing clusters, and predicts the energy consumption and duration of each task to be processed by the benchmark cluster through the metadata differences between the multiple historical processing tasks and the multiple tasks to be processed, to obtain a benchmark energy consumption and a benchmark duration; converts the energy consumption of each first cluster according to the benchmark energy consumption and the energy consumption proportional factor, obtains the corresponding first energy consumption, and calculates the energy consumption proportional factor according to the benchmark duration; length and duration proportional factor, convert the duration of each first cluster to obtain the corresponding first duration; input the benchmark energy consumption and benchmark duration corresponding to the benchmark cluster, as well as the first energy consumption and first duration corresponding to each first cluster into the preset target model to obtain the model to be solved, and the model to be solved includes a comprehensive energy consumption index under the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters; based on the model to be solved, minimize the comprehensive energy consumption index, determine the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters, and allocate multiple tasks to be processed to multiple intelligent computing clusters according to the allocation relationship. In this way, a benchmark cluster can be selected as a standardized reference point, so that the energy consumption and duration of different intelligent computing clusters can be compared under the same standard, so as to achieve fast and accurate conversion of energy consumption and duration; and, by accurately predicting the energy consumption and execution time of tasks on clusters in different regions, the uncertainty caused by the actual operation of tasks can be avoided, the risk and cost of scheduling decisions can be reduced, and the comprehensive energy consumption index of the model to be solved can be minimized to find the best balance between energy consumption and task completion time, achieve optimal allocation of overall resources, avoid resource waste and environmental impact caused by polling allocation, and make computing resources reasonably used, which is conducive to the sustainable development of the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the architecture of a resource scheduling system based on an intelligent computing cluster provided in an embodiment of the present application;

[0049] Figure 2 It is a flow chart of a resource scheduling method based on an intelligent computing cluster provided in an embodiment of the present application;

[0050] Figure 3 It is a functional module diagram of a resource scheduling device based on an intelligent computing cluster provided in an embodiment of the present application;

[0051] Figure 4 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] An intelligent computing cluster is a computing resource and architecture that combines high-performance computing capabilities with intelligent technologies (such as artificial intelligence, big data analysis, etc.). It aims to improve the utilization of computing resources, reduce energy consumption, and accelerate the execution of complex computing tasks through intelligent management and optimization strategies.

[0056] However, the physical facilities or operating environments (i.e., data centers) used to host and run these intelligent computing clusters are energy-intensive, accounting for a significant proportion of the total electricity consumption in society. As the demand for computing power for artificial intelligence increases, the energy consumption problem is becoming increasingly prominent. Therefore, it is necessary to reduce the energy consumption and carbon emissions of intelligent computing clusters in order to provide powerful computing power for artificial intelligence technology while reducing the negative impact on the environment.

[0057] In related technologies, when tasks are assigned to clusters for processing, the order in which each intelligent computing cluster processes tasks is usually determined by the order in which the tasks are submitted. After the tasks are submitted to the scheduling system, they will be recorded and queued. The scheduler will allocate them to appropriate computing resources or clusters based on the order in which the tasks enter the queue. However, this allocation method makes it impossible to make reasonable use of computing resources, which is not conducive to the sustainable development of the environment.

[0058] Based on this, the embodiments of the present application provide a resource scheduling method, device, equipment and storage medium based on an intelligent computing cluster, which can make rational use of computing resources and is conducive to the sustainable development of the environment.

[0059] The resource scheduling method, apparatus, device and storage medium based on the intelligent computing cluster provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the resource scheduling system based on the intelligent computing cluster in the embodiments of the present application is described.

[0060] Please refer to Figure 1 In some implementations, an embodiment of the present application provides a resource scheduling system based on an intelligent computing cluster, including multiple intelligent computing clusters and a unified scheduler.

[0061] In some implementations, the pending tasks may be initiated by a terminal user or an application. The terminal user may be an individual user, an enterprise user, or other system user, who submits the pending tasks through various terminal devices (such as computers, mobile phones, etc.).

[0062] Exemplarily, the unified scheduler can be installed on a computer device, which can be a management node of a server, data center or cloud platform, and is responsible for collecting and managing resource information of all intelligent computing clusters, including the load, available resources, geographic location, unit power consumption and unit carbon consumption of computing nodes. After receiving the pending tasks, the unified scheduler can parse the task information, evaluate it according to the current resource status of each intelligent computing cluster, calculate the execution cost of each pending task on different intelligent computing clusters according to the preset optimization algorithm, perform global optimization, and assign the pending tasks to the most suitable intelligent computing cluster for execution.

[0063] The resource scheduling method based on the intelligent computing cluster in the embodiment of the present application can be illustrated by the following embodiment.

[0064] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for enabling the normal operation of the embodiment of the present application will be obtained.

[0065] In the embodiment of the present application, the resource scheduling device based on the intelligent computing cluster will be described from the perspective of the resource scheduling device based on the intelligent computing cluster, and the resource scheduling device based on the intelligent computing cluster can be specifically integrated in a computer device. Figure 2 , Figure 2 The flowchart of the steps of the resource scheduling method based on the intelligent computing cluster provided in the embodiment of the present application is as follows:

[0066] Step 101 , benchmark multiple intelligent computing clusters corresponding to multiple regions through a benchmark test program to obtain an energy consumption proportional factor and a duration proportional factor of each first cluster relative to the benchmark cluster.

[0067] In some embodiments, in order to be able to compare the performance and energy consumption of different intelligent computing clusters in a standardized manner and establish a standardized comparison basis, a benchmark cluster can be selected to obtain the energy consumption proportional factor and duration proportional factor of each first cluster relative to the benchmark cluster, so as to quantify the differences between different intelligent computing clusters and improve scheduling efficiency and allocation rationality.

[0068] Among them, a benchmark program can be a standardized testing tool or process for evaluating the performance of a computer system or component, which contains a series of predefined workloads and can be used to simulate a specific type of application scenario. For example, a benchmark program can be MLPerf, Netperf, etc.

[0069] Among them, an intelligent computing cluster can be a group of interconnected computers or servers used for intelligent computing, which work together to complete complex computing tasks, especially those artificial intelligence and machine learning tasks that require a lot of computing resources.

[0070] The first cluster may be all other intelligent computing clusters except the intelligent computing cluster selected as the benchmark. The first cluster will be compared with the benchmark cluster to determine their respective performance and energy consumption characteristics.

[0071] Among them, the benchmark cluster can be the intelligent computing cluster selected as the reference standard. All performance and energy consumption measurements will be compared based on the data of the benchmark cluster. The benchmark cluster can be selected according to the stability of each intelligent computing cluster or randomly. There is no limit on the selection method.

[0072] The energy consumption ratio factor may be a ratio of the energy consumption of an intelligent computing cluster to that of a benchmark cluster to complete the same task. For example, if the energy consumption ratio factor of intelligent computing cluster A is 1.2, it means that the energy consumption required to complete the same task is 20% more than that of the benchmark cluster.

[0073] The duration scaling factor may represent the ratio of the time required for an intelligent computing cluster to complete the same task relative to the benchmark cluster. For example, if the duration scaling factor of intelligent computing cluster B is 0.8, it means that the time required to complete the same task is 20% less than that of the benchmark cluster.

[0074] For example, an intelligent computing cluster with stable performance and high energy efficiency can be selected from multiple intelligent computing clusters as a benchmark cluster so that the performance and energy consumption characteristics of other intelligent computing clusters can be compared with it. All intelligent computing clusters are tested using a standardized benchmark test program (such as MLPerf), and the test duration and test energy consumption of each cluster are recorded.

[0075] Furthermore, for each first cluster, the ratio of its test energy consumption to the benchmark cluster can be calculated. Specifically, the energy consumption ratio factor of the first cluster = the test energy consumption of the first cluster / the test energy consumption of the benchmark cluster; the duration ratio factor of the first cluster = the test duration of the first cluster / the test duration of the benchmark cluster.

[0076] By calculating the energy consumption proportional factor and the duration proportional factor, it is convenient to combine the test results of the benchmark cluster and quickly predict the performance of the pending tasks on different intelligent computing clusters. Compared with scheduling or polling scheduling only within a single cluster, this method can support the scheduling system to make data-based decisions and provide a scientific basis for task scheduling.

[0077] In some implementations, in order to compare the performance and energy consumption of different intelligent computing clusters in a standardized manner, a benchmark cluster may be selected and benchmarked for all intelligent computing clusters under the same environment to establish a standard reference index, thereby eliminating the instability of directly comparing absolute performance and energy consumption indicators. This approach can eliminate individual differences and ensure the fairness and accuracy of the comparison results. For example, step 101 may include:

[0078] (101.1) Benchmark multiple intelligent computing clusters in a distributed cluster using a benchmark test program to obtain the test energy consumption and test duration of each intelligent computing cluster after running the benchmark test program;

[0079] (101.2) Determine a benchmark cluster and multiple first clusters from multiple intelligent computing clusters, and determine the energy consumption proportional factor of each first cluster relative to the benchmark cluster based on the ratio of the test energy consumption of each first cluster to the benchmark cluster, and determine the energy consumption proportional factor of each first cluster relative to the benchmark cluster based on the ratio of the test duration of each first cluster to the benchmark cluster.

[0080] The test energy consumption may be the total energy consumed by each intelligent computing cluster during the execution of the benchmark test program. The test energy consumption is an important indicator for evaluating the energy efficiency of the corresponding intelligent computing cluster.

[0081] The test duration may be the total time required for the corresponding intelligent computing cluster to complete the benchmark test program. The test duration is an important indicator for evaluating cluster computing performance.

[0082] For example, if there is a distributed intelligent computing cluster, including multiple intelligent computing clusters in multiple regions. First, an intelligent computing cluster with stable performance and high energy efficiency can be selected from all available distributed intelligent computing clusters as the benchmark cluster, for example, intelligent computing cluster A is selected as the benchmark cluster. Alternatively, the benchmark cluster can be selected after all intelligent computing clusters have completed the benchmark test, and the selection time is not limited.

[0083] Furthermore, MLPerf benchmarks can be run on benchmark cluster A and all other intelligent computing clusters. At the same time, it should be ensured that all tests are performed under the same conditions, including using the same input data, algorithm version, and environment settings, and recording the computing time and energy consumption of each intelligent computing cluster when executing the benchmark.

[0084] Further, after selecting the reference cluster and the first cluster, the energy consumption proportional factor of each first cluster relative to the reference cluster A can be calculated. For example, if the energy consumption of the first cluster B is 1000 kWh and the energy consumption of the reference cluster A is 800 kWh, then the energy consumption proportional factor of the first cluster B is 1000 / 800=1.25.

[0085] Furthermore, the duration ratio factor of each first cluster relative to the reference cluster A may be calculated. For example, if the calculation duration of the first cluster B is 120 minutes and the calculation duration of the reference cluster A is 100 minutes, the duration ratio factor of the first cluster B is 120 / 100=1.2.

[0086] Through the above methods, a comprehensive evaluation of the performance and energy consumption of distributed intelligent computing clusters can be achieved, providing scientific and efficient support for task scheduling, which helps to optimize resource utilization, save computing resources, and improve scheduling efficiency.

[0087] Step 102, obtain multiple pending tasks and multiple historical processing tasks of multiple intelligent computing clusters, and predict the energy consumption and duration of the benchmark cluster executing each pending task through the metadata differences between the multiple historical processing tasks and the multiple pending tasks to obtain the benchmark energy consumption and benchmark duration.

[0088] In some embodiments, in order to achieve rapid and accurate prediction of the energy consumption and duration of tasks to be processed in a distributed intelligent computing cluster, historical processing tasks similar to the metadata of the tasks to be processed can be determined to predict the baseline energy consumption and baseline duration of the benchmark cluster executing the tasks to be processed. This not only improves the efficiency and accuracy of the scheduling system, but also promotes the development of green computing.

[0089] Among them, the pending tasks may be computing tasks currently waiting to be assigned and executed, and these tasks have not yet been run on any intelligent computing cluster.

[0090] Among them, historical processing tasks can be computing tasks that have been completed and run on the intelligent computing cluster, and the energy consumption and duration data of these tasks have been recorded.

[0091] The metadata may include various attribute information related to the computing task, including user name, group name, task name, file name, CPU occupancy, GPU occupancy, NPU occupancy, task start command, etc.

[0092] The benchmark energy consumption may be a predicted energy consumption value when the task to be processed runs on the benchmark cluster.

[0093] The benchmark duration may be the predicted duration of the task to be processed running on the benchmark cluster.

[0094] In some embodiments, the metadata of each task to be processed can be collected, and the metadata, energy consumption information, and duration information of all completed historical processing tasks can be extracted from the monitoring log of the intelligent computing cluster. In order to ensure the authenticity and feasibility of the prediction, the metadata of the historical processing tasks should be the same as the metadata of the tasks to be processed, for example, both include user name, group name, task name, file name, CPU occupancy, GPU occupancy, NPU occupancy, task start command, etc.

[0095] Furthermore, the k-Nearest Neighbors (kNN) algorithm can be used to find several known tasks with metadata closest to the historical processing tasks based on the metadata characteristics of the task to be processed. For example, the Euclidean distance between the task to be processed and the metadata of each historical processing task can be calculated to obtain multiple historical processing tasks with the closest distance to the task to be processed (that is, the highest similarity). The number of historical processing tasks finally determined can be determined by oneself, such as 5, 10, etc.

[0096] Furthermore, since the historical processing tasks may not all be executed by the benchmark cluster, but may also be executed by other intelligent computing clusters, in order to ensure the uniformity of the prediction scale, the historical processing tasks of other intelligent computing clusters can be converted into energy consumption and duration based on the energy consumption proportional factor and the duration proportional factor, and the energy consumption and duration of the benchmark cluster when executing the historical processing tasks can be predicted. For example, the energy consumption of the benchmark cluster in processing the historical processing task can be obtained based on the ratio of the energy consumption of the historical processing task to the corresponding energy consumption proportional factor; the duration of the benchmark cluster in processing the historical processing task can be obtained based on the ratio of the duration of the historical processing task to the corresponding duration proportional factor.

[0097] Furthermore, after predicting the energy consumption and duration of the benchmark cluster processing the selected historical processing tasks, the energy consumption can be averaged or weighted averaged to predict the benchmark energy consumption of the benchmark cluster processing the pending tasks, and the duration can be averaged to predict the benchmark duration of the benchmark cluster processing the pending tasks.

[0098] The energy consumption and duration prediction method based on the distance between task metadata is sufficiently accurate in practical applications, especially in the absence of more detailed task characteristics. This method provides an effective and efficient solution. By continuously accumulating and optimizing the records of metadata and historical energy consumption data, the accuracy of energy consumption and duration prediction will gradually improve, thereby further improving the quality of scheduling decisions.

[0099] In some implementations, in order to more accurately calculate the metadata difference between the task to be processed and the historical processing task, so as to find the most similar historical processing task, the complex metadata features can be automatically processed by the pre-trained target model to reduce manual intervention and improve computing efficiency. For example, in step 102, "predicting the energy consumption and duration of each task to be processed by the benchmark cluster through the metadata differences between multiple historical processing tasks and multiple tasks to be processed, and obtaining the benchmark energy consumption and benchmark duration" may include:

[0100] (102.1) calculating the difference between the multiple first metadata corresponding to each task to be processed and the multiple second metadata corresponding to each historical processing task using the pre-trained target model to obtain a target difference value;

[0101] (102.2) determining a plurality of reference processing tasks from the plurality of historical processing tasks based on a plurality of target difference values ​​corresponding to the plurality of historical processing tasks;

[0102] (102.3) Obtain a reference energy consumption and a reference duration for each reference processing task, and predict the energy consumption of the benchmark cluster for processing the corresponding pending tasks based on multiple reference energy consumptions to obtain a benchmark energy consumption, and predict the duration of the benchmark cluster for processing the corresponding pending tasks based on multiple reference durations to obtain a benchmark duration.

[0103] The target model may be a pre-trained machine learning model, which is used to calculate the metadata difference between the task to be processed and the historical processing tasks, and to determine the most similar historical processing tasks.

[0104] Among them, the first metadata can be the metadata of the task to be processed, including multiple important attributes related to the task, such as user name, group name, task name, file name, central processing unit (CPU) occupancy number, graphics processing unit (GPU) occupancy number, neural network processor (NPU) occupancy number, task start command, etc.

[0105] Among them, the second metadata can be the metadata of the historical processing task, which also includes multiple important attributes related to the task, such as user name, group name, task name, file name, CPU occupancy, GPU occupancy, NPU occupancy, task start command, etc.

[0106] The target difference value may be a metadata difference between the task to be processed and the historical processing task calculated by the target model.

[0107] The reference processing tasks may be selected from multiple historical processing tasks based on the target difference value and are most similar to the task to be processed. The number of reference processing tasks selected may be determined according to actual conditions, such as 3, 5, 10, etc.

[0108] Among them, the reference energy consumption can be the actual energy consumption value of the reference processing task in the historical operation of the corresponding intelligent computing cluster. The reference energy consumption can be determined based on the product of the operation duration when the corresponding intelligent computing cluster processes the reference processing task and the average power.

[0109] The reference duration may be the operation duration when the corresponding intelligent computing cluster processes the reference processing task.

[0110] Exemplarily, if there are pending tasks T1 and T2, and historical processing tasks H1, H2, H3, and H4, take H1 as an example, calculate the target difference values ​​between the multiple first metadata of the pending task T1 and the multiple second metadata of H1. First, feature extraction is required. If the first metadata of T1 is: Username: user1, Task name: task1, CPU occupancy: 4, GPU occupancy: 1, NPU occupancy: 0, Task start command: python script1.py. The second metadata of H1 is: Username: user1, Task name: task3, CPU occupancy: 4, GPU occupancy: 1, NPU occupancy: 0, Task start command: pythonscript3.py. The target model can output the target difference value of T1 and H1 as 0.1 based on the metadata in T1 and H1.

[0111] Exemplarily, the target model can calculate the Euclidean distance between the task to be processed and the multiple second metadata of each historical processing task based on the multiple first metadata of the task to be processed, and then obtain the target difference value of T1 and H1. For example, the numerical vector of T1 is [1,1,4,1,0,1], and the numerical vector of H1 is [1,3,4,1,0,2]. By calculating the difference of metadata through Euclidean distance, the target difference value can be obtained to be approximately 2.236. Exemplarily, for user names and task names, you can use numbers or hashes to convert strings into numerical values, for example, user1 can be converted to 1, etc.; for task startup commands, they can also be encoded into numerical values, for example, python script1.py is encoded as 1, python script3.py is encoded as 3, etc., and the embodiments of the present application do not specifically limit this.

[0112] Similarly, the target difference values ​​between T1 and H2, H3, H4, and the target difference values ​​between T2 and H1, H2, H3, H4 can be determined. The steps are as mentioned above and will not be described in detail here.

[0113] Furthermore, for each task to be processed, after the target model outputs its target difference value with all historical processing tasks, the target difference values ​​can be sorted from small to large, and the k historical processing tasks with the smallest distance are selected as reference processing tasks, as the k "nearest neighbors" of the task to be processed. The value of k can be determined according to actual conditions, for example, 3, 7, 10, etc. can be selected.

[0114] Furthermore, the reference energy consumption and reference duration of each reference processing task can be obtained to predict the benchmark energy consumption and benchmark duration of the benchmark cluster processing tasks that have the same or similar metadata as the reference processing task. For example, the reference energy consumption of all selected reference processing tasks can be averaged to obtain the benchmark energy consumption, such as ,in, represents the baseline energy consumption, represents the reference energy consumption, Indicates the number of reference processing tasks. ,in, Indicates the base duration. Indicates the reference duration. Indicates the number of reference processing tasks.

[0115] In some implementations, the weight of each reference processing task may be determined according to the inverse ratio of the target difference value, so that the reference processing task with a smaller target difference value has a higher corresponding weight, and the reference processing task with a larger target difference value has a lower corresponding weight. The calculation formula is as follows:

[0116] ;

[0117] in, represents the reference energy consumption, express The target difference value between the corresponding reference processing task j and the task to be processed, Indicates the number of reference processing tasks.

[0118] Benchmark duration The calculation formula is as follows:

[0119] ;

[0120] in, Indicates the reference duration. express The target difference value between the corresponding reference processing task j and the task to be processed, Indicates the number of reference processing tasks.

[0121] It should be noted that since the intelligent computing cluster that processes the reference processing task may not be the benchmark cluster, in order to ensure the accuracy of the prediction, the reference energy consumption can be converted according to a pre-set reference energy consumption ratio factor to predict the energy consumption of the benchmark cluster when processing the reference processing task, and the reference energy consumption can be converted according to a pre-set reference duration ratio factor to predict the duration of the benchmark cluster processing the reference processing task.

[0122] In some embodiments, not only the benchmark energy consumption and benchmark duration of the benchmark cluster can be predicted through historical processing tasks, but also the energy consumption and duration of other intelligent computing clusters processing all tasks to be processed can be predicted through historical processing tasks. The specific method is that for each intelligent computing cluster, multiple historical processing tasks processed by the intelligent computing cluster are obtained, and multiple reference processing tasks with the smallest distance from the tasks to be processed are determined according to the target difference values, and the energy consumption and duration of the intelligent computing cluster processing the tasks to be processed are predicted based on the reference duration and reference energy consumption of the reference processing tasks.

[0123] Through the above methods, historical data can be used quickly and effectively to predict energy consumption and duration, optimize resource allocation, and improve the efficiency and accuracy of allocation, rather than spending huge amounts of time and computing power to analyze and allocate the configuration of the intelligent computing cluster. In addition, by continuously accumulating and optimizing historical data, the prediction accuracy can be further improved to support more efficient scheduling decisions.

[0124] In some implementations, since different intelligent computing clusters may have different hardware configurations, performance indicators, and energy efficiency characteristics, when the intelligent computing cluster that processes the reference processing task is not the benchmark cluster, it is necessary to convert the energy consumption and duration into the benchmark energy consumption and benchmark duration executed on the benchmark cluster to avoid prediction errors caused by differences in hardware configurations and energy efficiency characteristics of different intelligent computing clusters, so as to make the prediction results more accurate and reliable. For example, "based on multiple reference energy consumptions, predict the energy consumption of the benchmark cluster processing the corresponding task to be processed to obtain the benchmark energy consumption, and based on multiple reference durations, predict the duration of the benchmark cluster processing the corresponding task to be processed to obtain the benchmark duration" in (102.3) may include:

[0125] (102.3.1) For multiple reference energy consumptions, determine that the intelligent computing cluster that processes the corresponding reference processing tasks is the second cluster;

[0126] (102.3.2) Obtain a reference energy consumption proportional factor and a reference duration proportional factor of each second cluster relative to the reference cluster;

[0127] (102.3.3) Based on the reference energy consumption ratio factor, convert the reference energy consumption of the corresponding second cluster to predict the second energy consumption of the reference processing task corresponding to the benchmark cluster, and predict the energy consumption of the benchmark cluster for processing the corresponding pending task based on the multiple second energy consumptions to obtain the benchmark energy consumption of the benchmark cluster for processing the pending task;

[0128] (102.3.4) Based on the reference duration ratio factor, the reference duration of the corresponding second cluster is converted to predict the second duration of the reference processing task corresponding to the benchmark cluster, and based on multiple second durations, the duration of the benchmark cluster processing the corresponding pending tasks is predicted to obtain the benchmark duration of the benchmark cluster processing the pending tasks.

[0129] The second cluster may be an intelligent computing cluster that processes reference processing tasks, and the second cluster may have different hardware configurations, performance indicators, and energy efficiency characteristics from the reference cluster.

[0130] The second energy consumption may be an energy consumption value obtained when the reference cluster processes the corresponding reference processing task after converting the reference energy consumption of the second cluster through the reference energy consumption proportional factor.

[0131] The second duration may be a duration value obtained by converting the reference duration of the second cluster by using the reference duration ratio factor, when the benchmark cluster processes the corresponding reference processing task.

[0132] It should be noted that in the above text, the energy consumption proportional factor and time proportional factor of each intelligent computing cluster in the distributed intelligent computing cluster relative to the benchmark cluster are introduced. Here, the preset energy consumption proportional factor and time proportional factor can be directly used. For the convenience of distinction and description, the reference energy consumption proportional factor and reference time proportional factor are used for naming.

[0133] In some implementations, if the reference processing task is run on a non-benchmark cluster, the reference energy consumption of the reference processing task needs to be converted into a second energy consumption, and the reference duration needs to be converted into a second duration. For example, the conversion formula is: second energy consumption conversion formula: second energy consumption = reference energy consumption of the reference target cluster / reference energy consumption ratio factor; second duration conversion formula: second duration = reference duration of the reference target cluster / reference duration ratio factor.

[0134] Furthermore, after converting the second energy consumption and second duration of all reference processing tasks corresponding to the task to be processed, the benchmark energy consumption can be obtained by adding and averaging all the second energy consumptions; and the benchmark duration can be obtained by adding and averaging all the second durations.

[0135] Alternatively, based on the target difference value corresponding to each reference processing task, all second energy consumptions may be weighted averaged to obtain a benchmark energy consumption; based on the target difference value corresponding to each reference processing task, all second durations may be weighted averaged to obtain a benchmark duration. When the target difference value is small, a larger weight is given to the corresponding second energy consumption or second duration, such as 0.4; when the target difference value is large, a smaller weight is given to the corresponding second energy consumption or second duration, such as 0.1, and so on. The specific calculation formula has been expanded above and will not be repeated here.

[0136] By converting the energy consumption and duration of the reference processing tasks, the accuracy and reliability of energy consumption and duration predictions between different intelligent computing clusters can be ensured.

[0137] Step 103: convert the energy consumption of each first cluster according to the reference energy consumption and the energy consumption proportional factor to obtain the corresponding first energy consumption; convert the duration of each first cluster according to the reference duration and the duration proportional factor to obtain the corresponding first duration.

[0138] In some embodiments, in order to improve the efficiency and accuracy of task scheduling, the energy consumption and duration of all first clusters processing each task to be processed can be converted. In this way, there is no need to perform actual operation tests on each task to be processed on each intelligent computing cluster. Instead, it can be estimated based on the data of the determined benchmark cluster and the pre-calibrated energy consumption proportional factors and duration proportional factors of other first clusters, thereby saving a lot of time and computing resources and enabling computing resources to be reasonably applied.

[0139] The first energy consumption may be an estimated energy consumption when the first cluster processes the task to be processed, and the first energy consumption may be obtained by converting the reference energy consumption through an energy consumption proportional factor corresponding to the first cluster.

[0140] The first duration may be an estimated duration when the first cluster processes the task to be processed, and the first duration may be obtained by converting the reference duration through a duration proportional factor corresponding to the first cluster.

[0141] Specifically, the first energy consumption = benchmark energy consumption × energy consumption ratio factor, and the first duration = benchmark duration × duration ratio factor. For example, the benchmark energy consumption of benchmark cluster A for processing pending task 1 is 800 kWh (kWh is kilowatt-hour), and the running time is 100 minutes. The energy consumption and duration of other first clusters processing pending task 1 can be predicted by the benchmark energy consumption and energy consumption ratio factors of different first clusters.

[0142] For example, if the energy consumption ratio factors of the first clusters B, C, and D relative to the benchmark cluster A are 1.25, 0.9, and 1.1, respectively, and the duration ratio factors of the first clusters B, C, and D relative to the benchmark cluster A are 1.2, 0.8, and 1.0, respectively. Then, it can be calculated that the first energy consumption of the first cluster B processing the pending task 1 is 800×1.25=1000kWh, and the first duration is 100×1.2=120 minutes; the first energy consumption of the first cluster C processing the pending task 1 is 800×0.9=720kWh, and the first duration is 100×0.8=80 minutes; the first energy consumption of the first cluster D processing the pending task 1 is 800×1.1=880kWh, and the first duration is 100×1.0=100 minutes.

[0143] The energy consumption proportional factor and duration proportional factor determined after the benchmark test program test can effectively convert the energy consumption and duration of the benchmark cluster to other first clusters, so as to quickly and accurately estimate the performance of different first clusters in processing pending tasks without actual testing, avoiding the large amount of computing resources and time costs consumed in measuring each pending task on each intelligent computing cluster, and improving the efficiency and accuracy of task scheduling.

[0144] Step 104, input the benchmark energy consumption and benchmark duration corresponding to the benchmark cluster, and the first energy consumption and first duration corresponding to each first cluster into a preset target model to obtain a model to be solved, which includes a comprehensive energy consumption index under the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters.

[0145] In some embodiments, in order to consider factors of multiple dimensions when scheduling tasks, the energy consumption and duration of each task to be processed can be input into the target model through the intelligent computing cluster to form a multi-objective problem and quantify the multi-objective problem into a specific mathematical function, so as to convert the complex task scheduling problem into a solvable optimization problem, improve the resource utilization efficiency of the entire system, and promote sustainable development.

[0146] The target model can be a mathematical model used to describe and quantify the distribution relationship between multiple pending tasks and multiple intelligent computing clusters in a distributed intelligent computing cluster. The target model comprehensively considers multiple factors such as power consumption, carbon emissions, task running time, hardware constraints, etc., aiming to find the optimal task allocation solution. The benchmark energy consumption, benchmark duration, first energy consumption, and first duration in the target model are all initialized values.

[0147] The model to be solved can be a specific mathematical model generated by inputting the baseline energy consumption and baseline duration of the baseline cluster, and the first energy consumption and first duration of each first cluster into the target model. The model to be solved includes the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters, and is used to solve the optimal task allocation solution.

[0148] Among them, the comprehensive energy consumption index can be a comprehensive indicator used to quantify the overall energy consumption of tasks when running on different intelligent computing clusters. The comprehensive energy consumption index combined with the constraint conditions can obtain the model to be solved. The comprehensive energy consumption index takes into account factors such as power consumption, carbon emissions, and task running time, and is calculated by weighted summation.

[0149] In some implementations, after the baseline energy consumption and baseline duration of the baseline cluster processing each task to be processed are calculated, a conversion can be performed based on the energy consumption proportional factor and the duration proportional factor to determine the first energy consumption and first duration of each first cluster processing each task to be processed. The conversion process has been introduced above and will not be repeated here. Below, the process of inputting the baseline energy consumption and baseline duration corresponding to the baseline cluster, and the first energy consumption and first duration corresponding to each first cluster into the preset target model to obtain the model to be solved is introduced.

[0150] Specifically, the power consumption index can be obtained by multiplying the energy consumption of each intelligent computing cluster processing the corresponding tasks to be processed by the unit power consumption, and the carbon emission index (i.e., carbon dioxide emission index) can be obtained by multiplying the energy consumption of each intelligent computing cluster processing the corresponding tasks to be processed by the unit carbon consumption. It should be noted that the early division of the intelligent computing cluster into the benchmark cluster and the first cluster is to improve the efficiency of predicting energy consumption and duration. When allocating tasks, they are all collectively referred to as intelligent computing clusters, the benchmark energy consumption and the first energy consumption are collectively referred to as cluster energy consumption, and the benchmark duration and the first duration are collectively referred to as cluster duration.

[0151] Exemplarily, the calculation formula of the power consumption index is as follows:

[0152] Power Consumption Index = ;

[0153] in, is the unit power consumption, The cluster energy consumption of intelligent computing cluster j for processing the corresponding tasks to be processed.

[0154] The calculation formula of carbon emission index is as follows:

[0155] Carbon Emission Index = ;

[0156] in, is the unit carbon consumption, The cluster energy consumption of intelligent computing cluster j for processing the corresponding tasks to be processed.

[0157] Furthermore, the power consumption index and carbon emission index of the tasks to be processed assigned to each intelligent computing cluster can be calculated based on the allocation relationship between the current multiple pending tasks and the multiple intelligent computing clusters. The power consumption indices of all intelligent computing clusters can be added up and the weights can be adjusted to obtain a power consumption function. The carbon emission indices of all intelligent computing clusters can be added up and the weights can be adjusted to obtain a carbon emission function. The cluster time of all intelligent computing clusters can be added up and the weights can be adjusted to obtain a time occupancy function.

[0158] For example, in the electricity consumption function, the carbon emission function and the time occupancy function, the binary variable To indicate the Is the pending task in If the task to be processed is on the intelligent computing cluster If it is run on ;otherwise, Since each pending task can only be assigned to one intelligent computing cluster to run, there is For all Established.

[0159] Exemplary, power consumption function The form can be as follows:

[0160] ;

[0161] in, It is The unit power consumption of the area where the intelligent computing cluster is located, that is, the unit price of electricity. It is To-be-processed tasks in the intelligent computing cluster Energy consumption of clusters running on .

[0162] For example, the carbon emission function The form can be as follows:

[0163] ;

[0164] in, It is The unit carbon consumption of the region where the intelligent computing cluster is located, that is, The CO2 emission factor of electricity for a smart computing cluster, It is To-be-processed tasks in the intelligent computing cluster Energy consumption of clusters running on .

[0165] Exemplary, duration occupancy function The form can be as follows:

[0166] ;

[0167] in, It is To-be-processed tasks in the intelligent computing cluster The cluster duration on .

[0168] Furthermore, by adding the power consumption function, carbon emission function, and time occupancy function, a comprehensive energy consumption index can be obtained. It is the weighted sum of electricity consumption function, carbon emission function and time occupancy function:

[0169] ;

[0170] Furthermore, the comprehensive energy consumption index and the corresponding constraint model can form a model to be solved. The constraint model can be the hardware constraint for each task to be processed, the number of nodes occupied by the intelligent computing cluster when processing the task to be processed does not exceed the total number of nodes in the intelligent computing cluster, etc., which can be set according to actual conditions. By trying to assign each task to be processed to a different intelligent computing cluster, the final task assignment can minimize the comprehensive energy consumption index while satisfying the constraint model.

[0171] Through the comprehensive optimization of multiple factors such as electricity consumption, carbon emissions and operating time, a balance between electricity cost, carbon emissions and operating time is achieved. In this way, not only a single factor is considered, but also multiple key factors are integrated, making the utilization of resources more reasonable and efficient, which is conducive to the sustainable development of the environment.

[0172] In some implementations, in order to achieve efficient, accurate and optimized scheduling of tasks in a distributed computing cluster, multiple factors such as power consumption, carbon emissions and task runtime can be comprehensively considered to ensure that the task allocation scheme is optimal in multiple dimensions, so as to improve resource utilization efficiency, reduce energy consumption and carbon emissions, and improve system performance. For example, step 104 may include:

[0173] (104.1) Determine the power consumption function and carbon emission function of the target model according to the baseline energy consumption and the first energy consumption input into a preset target model;

[0174] (104.2) Determine a duration occupancy function of the target model according to the reference duration and the first duration input into a preset target model;

[0175] (104.3) Based on the electricity consumption function, carbon emission function and time occupancy function, the model to be solved is obtained.

[0176] The power consumption function can be used to quantify the total power consumption when multiple pending tasks are run on the corresponding multiple intelligent computing clusters. The power consumption function is calculated by weighting and summing the power consumption of multiple pending tasks processed by multiple intelligent computing clusters and then adjusting the weights.

[0177] The carbon emission function can be used to quantify the total carbon emissions of multiple pending tasks when they are run on the corresponding multiple intelligent computing clusters. The carbon emission function is calculated by weighting and summing the carbon emissions of multiple pending tasks processed by multiple intelligent computing clusters and then adjusting the weights.

[0178] The duration occupancy function can be used to quantify the total duration of multiple pending tasks when they are executed on the corresponding multiple intelligent computing clusters. The duration occupancy function is calculated by weighting and summing the durations of multiple pending tasks processed by multiple intelligent computing clusters and then adjusting the weights.

[0179] For example, in the target model, an initial value may be set for cluster energy consumption and cluster duration. At this time, the calculated energy consumption and duration of each intelligent computing cluster processing each task to be processed are input into the target model, and the target model's power consumption function, carbon emission function and duration occupancy function can be updated to obtain the model to be solved.

[0180] For example, in the electricity consumption function, the carbon emission function and the time occupancy function, the binary variable To express the Is the pending task in If the task to be processed is on the intelligent computing cluster If it is run on ;otherwise, Since each pending task can only be assigned to one intelligent computing cluster to run, there is For all Established.

[0181] Exemplary, power consumption function The form can be as follows:

[0182] ;

[0183] in, It is The unit power consumption of the area where the intelligent computing cluster is located, that is, the unit price of electricity. It is To-be-processed tasks in the intelligent computing cluster Energy consumption of clusters running on .

[0184] For example, the carbon emission function The form can be as follows:

[0185] ;

[0186] in, It is The unit carbon consumption of the region where the intelligent computing cluster is located, that is, The CO2 emission factor of electricity for a smart computing cluster, It is To-be-processed tasks in the intelligent computing cluster Energy consumption of clusters running on .

[0187] Exemplary, duration occupancy function The form can be as follows:

[0188] ;

[0189] in: It is To-be-processed tasks in the intelligent computing cluster The cluster duration on .

[0190] Furthermore, by adding the power consumption function, carbon emission function, and time occupancy function, a comprehensive energy consumption index can be obtained. It is the weighted sum of electricity consumption function, carbon emission function and time occupancy function:

[0191] ;

[0192] Furthermore, the comprehensive energy consumption index and the corresponding constraint model can form a model to be solved. The constraint model can be the hardware constraint for each task to be processed, the number of nodes occupied by the intelligent computing cluster when processing the task to be processed does not exceed the total number of nodes in the intelligent computing cluster, etc., which can be set according to actual conditions. By trying to assign each task to be processed to a different intelligent computing cluster, the final task assignment can minimize the comprehensive energy consumption index while satisfying the constraint model.

[0193] For example, the unit electricity consumption and unit carbon consumption can be obtained from the energy management department, or can be obtained by self-measurement. This application does not limit the method of obtaining.

[0194] Through multi-dimensional comprehensive optimization, efficient, accurate and flexible task scheduling of distributed intelligent computing clusters is achieved, which helps to reduce energy consumption and carbon emissions and improve system performance and resource utilization.

[0195] For example, in order to achieve multi-dimensional comprehensive optimization and ensure that the task allocation scheme is optimal in multiple aspects, a target model can be constructed and the tasks to be processed can be assigned to different intelligent computing clusters to minimize the comprehensive energy consumption index. For example, the target model is constructed in the following way:

[0196] (A.1) Obtain the initial power consumption index, initial carbon emission index and initial duration occupancy index of each intelligent computing cluster processing the initial assigned initial tasks;

[0197] (A.2) obtaining an initial power consumption function according to the sum of the initial power consumption indexes corresponding to the multiple intelligent computing clusters and the product of the preset first weight;

[0198] (A.3) Obtaining an initial carbon emission function based on the sum of the initial carbon emission indexes corresponding to the multiple intelligent computing clusters and the product of the preset second weight;

[0199] (A.4) obtaining an initial duration occupancy function according to the sum of multiple initial duration occupancy indexes corresponding to multiple intelligent computing clusters and the product of a preset third weight; wherein the first weight, the second weight and the third weight are determined according to the resource emphasis condition;

[0200] (A.5) Obtain a constraint model constructed by the quantity constraint function and hardware constraint function set for each intelligent computing cluster, and construct a target model based on the initial power consumption function, initial carbon emission function, initial time occupancy function and the constraint model.

[0201] Among them, the initial power consumption index can be used to quantify the power consumption of each intelligent computing cluster when processing the initial task. The initial power consumption index can be determined based on the product of the initial power consumption value and the unit power consumption when the corresponding intelligent computing cluster processes the corresponding initial task. The initial power consumption value can be a randomly set non-zero initial value.

[0202] Among them, the initial carbon emission index can be used to quantify the carbon emissions of each intelligent computing cluster when processing the initial task. The initial carbon emission index can be determined based on the product of the initial carbon emission value and the unit carbon consumption when the corresponding intelligent computing cluster processes the corresponding initial task. The initial carbon emission value can be a randomly set non-zero initial value.

[0203] The initial duration occupancy index can be used to quantify the running time of each intelligent computing cluster when processing the initial task. The initial duration occupancy index can be a randomly set non-zero initial value.

[0204] The first weight may be used to weight the importance of power consumption in the comprehensive energy consumption index.

[0205] Among them, the initial power consumption function can be used to quantify the total power consumption when multiple intelligent computing clusters process corresponding multiple initial tasks.

[0206] The second weight may be used to weight the importance of carbon emissions in the comprehensive energy consumption index.

[0207] Among them, the initial carbon emission function can be used to quantify the total carbon emissions when multiple intelligent computing clusters process corresponding multiple initial tasks.

[0208] The third weight may be used to weight the importance of duration occupancy in the comprehensive energy consumption index.

[0209] Among them, the initial duration occupancy function can be used to quantify the total running time when multiple intelligent computing clusters process corresponding multiple initial tasks.

[0210] The resource emphasis condition may be the degree of emphasis on different resources (power consumption, carbon emissions, and time occupation) during the optimization process. By adjusting the first weight, the second weight, and the third weight, the emphasis of the optimization target may be changed.

[0211] Among them, the constraint model can be used to ensure that the task allocation plan meets specific constraints, such as each pending task can only be assigned once, the number of intelligent computing clusters assigned to each pending task cannot be greater than one, the hardware resources of the intelligent computing cluster cannot be overloaded, and there are pending tasks that need to be run on specific hardware.

[0212] For example, in the initial power consumption function, the initial carbon emission function and the initial time occupation function, the binary variable To express the Is the pending task (or the randomly assigned initial task) in the If the task to be processed is on the intelligent computing cluster If it is run on ;otherwise, Since each pending task can only be assigned to one intelligent computing cluster to run, there is For all Established.

[0213] In some implementations, the initial cluster energy consumption and cluster duration may be a randomly set non-zero initial value, or may be a random value. Express the initial cluster energy consumption using The initial cluster duration is expressed. After the cluster energy consumption (including the baseline energy consumption and the first energy consumption) and the cluster duration (including the baseline duration and the first duration) of each intelligent computing cluster for processing each task to be processed are calculated, the cluster energy consumption and the cluster duration can be input into the target model to update the model to be solved. In some embodiments, after the cluster energy consumption and the cluster duration are input into the target model, the unit power consumption and the unit carbon consumption corresponding to the intelligent computing cluster can be updated together.

[0214] Exemplary, initial power consumption function The form can be as follows:

[0215] ;

[0216] in, It is The unit power consumption of the area where the intelligent computing cluster is located, that is, the unit price of electricity. It is To-be-processed tasks in the intelligent computing cluster The initial cluster energy consumption value running on the , when the cluster energy consumption is calculated, the initial cluster energy consumption value is updated accordingly, is the corresponding initial power consumption index.

[0217] Exemplary, initial carbon emission function The form can be as follows:

[0218] ;

[0219] in, It is The unit carbon consumption of the region where the intelligent computing cluster is located, that is, The initial carbon dioxide emission factor of electricity for a smart computing cluster, It is To-be-processed tasks in the intelligent computing cluster The initial cluster energy consumption value running on the , when the cluster energy consumption is calculated, the initial cluster energy consumption value is updated accordingly, is the corresponding initial carbon emission index.

[0220] Exemplarily, the initial duration occupies the function The form can be as follows:

[0221] ;

[0222] in, It is To-be-processed tasks in the intelligent computing cluster The initial duration occupancy value on (also called the initial duration occupancy index).

[0223] Furthermore, the initial comprehensive energy consumption index can be obtained by summing the initial power consumption function, the initial carbon emission function, and the initial duration occupancy function. It is the weighted sum of the initial electricity consumption function, the initial carbon emission function and the initial duration occupancy function:

[0224] ;

[0225] in, is the first weight, is the second weight, The third weight.

[0226] In some implementations, the first weight, the second weight, and the third weight may be determined according to resource emphasis conditions. For example, in a high energy consumption environment, the first weight, the second weight, and the third weight may be increased. To give priority to reducing electricity costs; under the requirement of low carbon emissions, it can be increased , which is beneficial to the sustainable development of the environment. The specific settings of the first weight, the second weight, and the third weight can also be determined according to the sensitivity of the tasks to be processed. For example, if the tasks to be processed are generally sensitive to electricity costs and hope to reduce electricity consumption, a higher first weight can be set. Similarly, if environmental protection is very important and carbon emissions are minimized, a higher second weight can be set. If the task completion time is very sensitive and the task is to be completed as soon as possible, a higher third weight can be set.

[0227] In some implementations, if there are specific requirements for individual tasks to be processed, for example, requiring that the power cost of task A to be processed be reduced as much as possible, the requirement may be added to the constraint model.

[0228] Furthermore, the initial comprehensive energy consumption index and the corresponding constraint model can form a target model. The constraint model can be the hardware constraint for each task to be processed, the number of nodes occupied by the intelligent computing cluster when processing the task to be processed does not exceed the total number of nodes in the intelligent computing cluster, etc., which can be set according to actual conditions. For example, the constraint model can be set to include a quantity constraint function: , for each intelligent computing cluster Established. Among them, Indicates pending tasks The number of nodes occupied, Indicates the intelligent computing cluster that processes the pending task k The capacity of the intelligent computing cluster is the total number of nodes. In addition, the constraint model can also include hardware constraint functions, for example, It means that the task 1 to be processed can only be run on the intelligent computing cluster 3 or the intelligent computing cluster 4, or the specific hardware of the task to be processed can also be constrained, and so on.

[0229] By setting the first weight, the second weight and the third weight to flexibly adjust the weight of each function, it can adapt to different business scenarios and resource conditions, ensure the scientificity and rationality of the optimization process, realize the efficient, accurate and flexible task scheduling of distributed intelligent computing clusters, and improve system performance and resource utilization.

[0230] In some implementations, in order to ensure that the task scheduling scheme is optimal in multiple dimensions, when designing the target model, expressions for the initial power consumption index, the initial carbon emission index, and the initial duration occupancy index may be designed to provide a scientific basis for optimizing task allocation and improve the rationality of resource scheduling. For example, (A.1) may include:

[0231] (A.1.1) Obtain the initial cluster energy consumption value and initial duration occupancy value of the initial task processed by each intelligent computing cluster;

[0232] (A.1.2) Obtain the unit power consumption corresponding to the area where each intelligent computing cluster is located, and determine the initial power consumption index of each intelligent computing cluster when processing the initial task based on the product of the initial cluster energy consumption value and the corresponding unit power consumption;

[0233] (A.1.3) Obtain the unit carbon consumption corresponding to the region where each intelligent computing cluster is located, and determine the initial carbon emission index of each intelligent computing cluster when processing the initial task based on the product of the initial cluster energy consumption value and the corresponding unit carbon consumption;

[0234] (A.1.4) Based on the initial duration occupancy value, determine the initial duration occupancy index of each intelligent computing cluster when processing the initial task.

[0235] The initial cluster energy consumption value is the energy value consumed by each intelligent computing cluster when processing the initial task. The initial cluster energy consumption value can be a randomly set non-zero initial value, or an actual energy consumption value obtained through historical data or actual measurement. The initial cluster energy consumption value can also be expressed in symbols. When the baseline energy consumption and the first energy consumption are calculated, the initial cluster energy consumption value is updated to a specific value.

[0236] The initial duration occupancy value may be the running time required by each intelligent computing cluster to process the initially assigned pending tasks. Since the model to be solved needs to be re-acquired and optimized later, the initial duration occupancy value may also be a random non-zero value.

[0237] The unit power consumption may be the unit price of power consumption in the area where each intelligent computing cluster is located, that is, the unit price of power cost.

[0238] The unit carbon consumption may be the carbon dioxide emission factor of electricity, that is, the amount of carbon dioxide emitted for each unit of electricity consumed.

[0239] Exemplarily, the calculation formula of the initial power consumption index is as follows:

[0240] Initial Power Consumption Index = ;

[0241] in, is the unit power consumption, The initial cluster energy consumption value of the intelligent computing cluster j for processing the corresponding tasks to be processed.

[0242] The calculation formula of the initial carbon emission index is as follows:

[0243] Initial carbon emission index = ;

[0244] in, is the unit carbon consumption, The initial cluster energy consumption value of the intelligent computing cluster j for processing the corresponding tasks to be processed.

[0245] For example, the initial duration occupancy index is the initial duration occupancy value, which can be preset to 0 or represented by the symbol express.

[0246] Through the above method, the initial power consumption index, initial carbon emission index and initial time occupancy index of each intelligent computing cluster when processing the initial task can be accurately quantified, which provides a scientific basis for optimizing task allocation, helps to reduce energy consumption and carbon emissions, and improve system performance and resource utilization.

[0247] Step 105 , performing minimization solution for the comprehensive energy consumption index based on the model to be solved, determining the allocation relationship between the multiple tasks to be processed and the multiple intelligent computing clusters, and allocating the multiple tasks to be processed to the multiple intelligent computing clusters according to the allocation relationship.

[0248] In some embodiments, in order to ensure the optimality of the task allocation scheme in multiple dimensions, different tasks to be processed can be assigned to different intelligent computing clusters, and the comprehensive energy consumption index of the model to be solved can be minimized by adjusting the allocation, thereby improving resource utilization efficiency and reducing energy consumption and carbon emissions.

[0249] For example, assume that there are three intelligent computing clusters (cluster B, cluster C, cluster D) and four pending tasks (task 1, task 2, task 3, task 4). Now it is necessary to allocate the four pending tasks to different intelligent computing clusters so as to minimize the comprehensive energy consumption index.

[0250] For example, the unit power consumption and unit carbon consumption of the intelligent computing cluster can be obtained first. For example, the unit power consumption of cluster B is The unit carbon consumption is 0.15 yuan / kWh (kWh is kilowatt-hour). 0.05kg / kWh, unit power consumption of cluster C The unit carbon consumption is 0.12 yuan / kWh. 0.04kg / kWh, unit power consumption of cluster D The unit carbon consumption is 0.14 yuan / kWh. 0.05kg / kWh. In which, kg / kWh is used to express the mass of carbon dioxide (CO2) emitted for every kilowatt-hour (kWh) of electricity generated or consumed, in kilograms (kg).

[0251] For example, after calculation, it can be obtained that the cluster energy consumption of cluster B for processing task 1 is 150 (for the convenience of description, subsequent units are omitted), and the cluster duration is 120; the cluster energy consumption for processing task 2 is 112.5, and the cluster duration is 108; the cluster energy consumption for processing task 3 is 131.25, and the cluster duration is 132; the cluster energy consumption for processing task 4 is 131.25, and the cluster duration is 132.

[0252] For example, after calculation, it can be obtained that the cluster energy consumption of cluster C for processing task 1 is 86.4, and the cluster duration is 80; the cluster energy consumption for processing task 2 is 64.8, and the cluster duration is 72; the cluster energy consumption for processing task 3 is 75.6, and the cluster duration is 88; the cluster energy consumption for processing task 4 is 97.2, and the cluster duration is 90.

[0253] For example, after calculation, it can be obtained that the cluster energy consumption of cluster D for processing task 1 is 123.2, and the cluster duration is 100; the cluster energy consumption for processing task 2 is 92.4, and the cluster duration is 90; the cluster energy consumption for processing task 3 is 107.8, and the cluster duration is 100; the cluster energy consumption for processing task 4 is 138.6, and the cluster duration is 100.

[0254] For example, the power consumption index of cluster B processing task 1 is calculated to be 150*0.15=22.5 yuan, and the carbon emission index is calculated to be 150*0.05=7.5 kg. (kg Kilograms of carbon dioxide represents the mass of carbon dioxide measured in kilograms, kg is kilograms, is carbon dioxide), the duration occupancy index is 120 minutes (refer to the above for specific acquisition methods); the power consumption index of cluster B processing task 2 is 16.875 yuan, and the carbon emission index is 5.625kg The duration occupancy index is 108 minutes; the power consumption index of cluster B processing task 3 is 19.6875 yuan, and the carbon emission index is 6.5625 kg The duration occupancy index is 132 minutes; the power consumption index of cluster D processing task 4 is 25.3125 yuan, and the carbon emission index is 8.4375 kg , the duration occupancy index is 132 minutes.

[0255] Similarly, the power consumption index, carbon emission index and duration index of cluster C and cluster D for processing task 1, task 2, task 3 and task 4 respectively can be calculated, which will not be described in detail here.

[0256] Furthermore, if the first weight is set to 0.5, the second weight is set to 0.3, and the third weight is set to 0.2, task 1 is assigned to cluster C, task 2 is assigned to cluster C, task 3 is assigned to cluster D, and task 4 is assigned to cluster B, then the final electricity consumption is 29.27425 yuan and the final carbon emissions are 5.96265 kg. , the duration is 76.8 minutes, and the comprehensive energy consumption index is the sum of the three, which is 112.0369. The model to be solved can be continuously optimized to minimize the comprehensive energy consumption index while satisfying the constraint model, so as to obtain the best allocation plan for each task to be processed to the corresponding intelligent computing cluster when minimizing the comprehensive energy consumption index.

[0257] The embodiment of the present application performs a benchmark test on multiple intelligent computing clusters corresponding to multiple regions through a benchmark test program, obtains an energy consumption proportional factor and a duration proportional factor of each first cluster relative to a benchmark cluster; obtains multiple tasks to be processed and multiple historical processing tasks of multiple intelligent computing clusters, and predicts the energy consumption and duration of each task to be processed by the benchmark cluster through the metadata differences between the multiple historical processing tasks and the multiple tasks to be processed, to obtain a benchmark energy consumption and a benchmark duration; converts the energy consumption of each first cluster according to the benchmark energy consumption and the energy consumption proportional factor, obtains the corresponding first energy consumption, and calculates the energy consumption proportional factor according to the benchmark duration; length and duration proportional factor, convert the duration of each first cluster to obtain the corresponding first duration; input the benchmark energy consumption and benchmark duration corresponding to the benchmark cluster, as well as the first energy consumption and first duration corresponding to each first cluster into the preset target model to obtain the model to be solved, and the model to be solved includes a comprehensive energy consumption index under the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters; based on the model to be solved, minimize the comprehensive energy consumption index, determine the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters, and allocate multiple tasks to be processed to multiple intelligent computing clusters according to the allocation relationship. In this way, a benchmark cluster can be selected as a standardized reference point, so that the energy consumption and duration of different intelligent computing clusters can be compared under the same standard, so as to achieve fast and accurate conversion of energy consumption and duration; and, by accurately predicting the energy consumption and execution time of tasks on clusters in different regions, the uncertainty caused by the actual operation of tasks can be avoided, the risk and cost of scheduling decisions can be reduced, and the comprehensive energy consumption index of the model to be solved can be minimized to find the best balance between energy consumption and task completion time, achieve optimal allocation of overall resources, avoid resource waste and environmental impact caused by polling allocation, and make computing resources reasonably used, which is conducive to the sustainable development of the environment.

[0258] See also Figure 3 The embodiment of the present application further provides a resource scheduling device based on an intelligent computing cluster, which can implement the above-mentioned resource scheduling method based on an intelligent computing cluster. The resource scheduling device based on an intelligent computing cluster includes:

[0259] A testing module 31 is used to perform a benchmark test on multiple intelligent computing clusters corresponding to multiple regions through a benchmark test program to obtain an energy consumption proportional factor and a duration proportional factor of each first cluster relative to the benchmark cluster;

[0260] The prediction module 32 is used to obtain multiple tasks to be processed and multiple historical processing tasks of multiple intelligent computing clusters, and predict the energy consumption and duration of each task to be processed by the benchmark cluster through the metadata differences between the multiple historical processing tasks and the multiple tasks to be processed, so as to obtain the benchmark energy consumption and benchmark duration;

[0261] The conversion module 33 is used to convert the energy consumption of each first cluster according to the reference energy consumption and the energy consumption proportional factor to obtain the corresponding first energy consumption, and to convert the duration of each first cluster according to the reference duration and the duration proportional factor to obtain the corresponding first duration;

[0262] An input module 34 is used to input the benchmark energy consumption and benchmark duration corresponding to the benchmark cluster, and the first energy consumption and first duration corresponding to each first cluster into a preset target model to obtain a model to be solved, wherein the model to be solved includes a comprehensive energy consumption index under the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters;

[0263] The solution module 35 is used to minimize the comprehensive energy consumption index based on the model to be solved, determine the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters, and allocate multiple tasks to be processed to multiple intelligent computing clusters according to the allocation relationship. The specific implementation method of the resource scheduling device based on the intelligent computing cluster is basically the same as the specific implementation method of the resource scheduling method based on the intelligent computing cluster mentioned above, which will not be repeated here. On the premise of meeting the requirements of the embodiments of the present application, the resource scheduling device based on the intelligent computing cluster can also be provided with other functional modules to implement the resource scheduling method based on the intelligent computing cluster in the above-mentioned embodiment.

[0264] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the resource scheduling method based on the intelligent computing cluster when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0265] See also Figure 4 , Figure 4 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0266] The processor 41 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0267] The memory 42 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 42 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 42, and the processor 41 calls and executes the resource scheduling method based on the intelligent computing cluster in the embodiment of this application;

[0268] Input / output interface 43, used to implement information input and output;

[0269] The communication interface 44 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0270] A bus 45 that transmits information between the various components of the device (e.g., the processor 41, the memory 42, the input / output interface 43, and the communication interface 44);

[0271] The processor 41 , the memory 42 , the input / output interface 43 and the communication interface 44 are connected to each other in communication within the device via a bus 45 .

[0272] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the resource scheduling method based on the intelligent computing cluster is implemented.

[0273] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0274] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0275] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0276] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0277] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0278] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0279] It should be understood that in the present application, "at least one (item)" and "several" refer to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0280] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0281] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0282] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0283] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0284] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A resource scheduling method based on intelligent computing cluster, characterized in that: The method comprises: Benchmark multiple intelligent computing clusters corresponding to multiple regions using a benchmark test program to obtain an energy consumption proportional factor and a duration proportional factor of each first cluster relative to the benchmark cluster; Acquire multiple tasks to be processed and multiple historical processing tasks of the multiple intelligent computing clusters, and predict the energy consumption and duration of each task to be processed by the benchmark cluster through metadata differences between the multiple historical processing tasks and the multiple tasks to be processed, so as to obtain benchmark energy consumption and benchmark duration; The method predicts the energy consumption and duration of each to-be-processed task executed by the benchmark cluster through the metadata differences between the multiple historical processing tasks and the multiple to-be-processed tasks to obtain the benchmark energy consumption and benchmark duration, including: calculating the metadata differences between the multiple first metadata corresponding to each to-be-processed task and the multiple second metadata corresponding to each historical processing task through a pre-trained target model to obtain a target difference value; determining multiple reference processing tasks from the multiple historical processing tasks based on the multiple target difference values ​​corresponding to the multiple historical processing tasks; obtaining the reference energy consumption and reference duration of each reference processing task, and predicting the energy consumption of the benchmark cluster processing the corresponding to-be-processed task based on the multiple reference energy consumptions to obtain the benchmark energy consumption, and predicting the duration of the benchmark cluster processing the corresponding to-be-processed task based on the multiple reference durations to obtain the benchmark duration; According to the reference energy consumption and the energy consumption proportional factor, convert the energy consumption of each first cluster to obtain a corresponding first energy consumption; according to the reference duration and the duration proportional factor, convert the duration of each first cluster to obtain a corresponding first duration; Input the benchmark energy consumption and the benchmark duration corresponding to the benchmark cluster, and the first energy consumption and the first duration corresponding to each first cluster into a preset target model to obtain a model to be solved, wherein the model to be solved includes a comprehensive energy consumption index under the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters; The comprehensive energy consumption index is minimized based on the model to be solved, an allocation relationship between the multiple tasks to be processed and the multiple intelligent computing clusters is determined, and the multiple tasks to be processed are allocated to the multiple intelligent computing clusters according to the allocation relationship.

2. The resource scheduling method based on intelligent computing cluster according to claim 1 is characterized in that: The step of inputting the benchmark energy consumption and the benchmark duration corresponding to the benchmark cluster, and the first energy consumption and the first duration corresponding to each first cluster into a preset target model to obtain a model to be solved includes: According to the benchmark energy consumption and the first energy consumption, input into a preset target model, determining the power consumption function and the carbon emission function of the target model; According to the reference duration and the first duration input into the preset target model, a duration occupancy function of the target model is determined; Based on the power consumption function, the carbon emission function and the time occupancy function, a model to be solved is obtained.

3. The resource scheduling method based on intelligent computing cluster according to claim 2 is characterized in that: The target model is constructed in the following way: Obtain the initial power consumption index, initial carbon emission index and initial duration occupancy index of each intelligent computing cluster processing the initial assigned initial tasks; Obtaining an initial power consumption function according to the product of the sum of the initial power consumption indexes corresponding to the multiple intelligent computing clusters and a preset first weight; Obtaining an initial carbon emission function according to the sum of the initial carbon emission indexes corresponding to the multiple intelligent computing clusters and the product of a preset second weight; An initial duration occupancy function is obtained according to the sum of the multiple initial duration occupancy indexes corresponding to the multiple intelligent computing clusters and the product of a preset third weight; wherein the first weight, the second weight and the third weight are determined according to a resource emphasis condition; Obtain a constraint model constructed by the quantity constraint function and the hardware constraint function set for each intelligent computing cluster, and construct a target model based on the initial power consumption function, the initial carbon emission function, the initial time occupancy function and the constraint model.

4. The resource scheduling method based on intelligent computing cluster according to claim 3 is characterized in that: The obtaining of the initial power consumption index, the initial carbon emission index and the initial duration occupancy index of the initial task initially assigned to each intelligent computing cluster includes: Obtain the initial cluster energy consumption value and initial duration occupancy value of each intelligent computing cluster processing the corresponding initial task; Obtaining the unit power consumption corresponding to the area where each intelligent computing cluster is located, and determining the initial power consumption index of each intelligent computing cluster when processing the initial task according to the product of the initial cluster energy consumption value and the corresponding unit power consumption; Obtaining the unit carbon consumption corresponding to the area where each intelligent computing cluster is located, and determining the initial carbon emission index when each intelligent computing cluster processes the initial task according to the product of the initial cluster energy consumption value and the corresponding unit carbon consumption; An initial duration occupancy index when each intelligent computing cluster processes the initial task is determined according to the initial duration occupancy value.

5. The resource scheduling method based on intelligent computing cluster according to claim 1, characterized in that: The benchmarking of the multiple intelligent computing clusters corresponding to the multiple regions by using a benchmarking program to obtain the energy consumption proportional factor and duration proportional factor of each first cluster relative to the benchmark cluster includes: Benchmarking multiple intelligent computing clusters in a distributed cluster using a benchmark test program to obtain the test energy consumption and test duration of each intelligent computing cluster after running the benchmark test program; Determine a benchmark cluster and multiple first clusters from the multiple intelligent computing clusters, and determine an energy consumption proportional factor of each first cluster relative to the benchmark cluster based on the ratio of the test energy consumption of each first cluster to the benchmark cluster, and determine the energy consumption proportional factor of each first cluster relative to the benchmark cluster based on the ratio of the test duration of each first cluster to the benchmark cluster.

6. The resource scheduling method based on intelligent computing cluster according to claim 1, characterized in that: The step of predicting the energy consumption of the benchmark cluster for processing the corresponding task to be processed based on multiple reference energy consumptions to obtain benchmark energy consumption, and predicting the duration of the benchmark cluster for processing the corresponding task to be processed based on multiple reference durations to obtain benchmark durations, includes: According to the plurality of reference energy consumptions, determining that the intelligent computing cluster that processes the corresponding reference processing tasks is the second cluster; Obtaining a reference energy consumption proportional factor and a reference duration proportional factor of each second cluster relative to the reference cluster; Based on the reference energy consumption proportional factor, convert the reference energy consumption of the corresponding second cluster to predict the second energy consumption of the reference processing task corresponding to the benchmark cluster, and predict the energy consumption of the benchmark cluster processing the corresponding task to be processed based on the multiple second energy consumptions to obtain the benchmark energy consumption when the benchmark cluster processes the task to be processed; Based on the reference duration ratio factor, the reference duration of the corresponding second cluster is converted to predict the second duration of the reference processing task corresponding to the benchmark cluster, and based on multiple second durations, the duration of the benchmark cluster processing the corresponding task to be processed is predicted to obtain the benchmark duration when the benchmark cluster processes the task to be processed.

7. A resource scheduling device based on intelligent computing cluster, characterized in that: The device comprises: A testing module, used to perform a benchmark test on multiple intelligent computing clusters corresponding to multiple regions through a benchmark test program to obtain an energy consumption proportional factor and a duration proportional factor of each first cluster relative to the benchmark cluster; A prediction module, used to obtain multiple tasks to be processed and multiple historical processing tasks of the multiple intelligent computing clusters, and predict the energy consumption and duration of the benchmark cluster to execute each task to be processed through the metadata differences between the multiple historical processing tasks and the multiple tasks to be processed, so as to obtain a benchmark energy consumption and a benchmark duration; wherein, the prediction of the energy consumption and duration of the benchmark cluster to execute each task to be processed through the metadata differences between the multiple historical processing tasks and the multiple tasks to be processed, so as to obtain a benchmark energy consumption and a benchmark duration, comprises: performing metadata difference calculations on multiple first metadata corresponding to each task to be processed and multiple second metadata corresponding to each historical processing task through a pre-trained target model, so as to obtain a target difference value; determining multiple reference processing tasks from the multiple historical processing tasks based on the multiple target difference values ​​corresponding to the multiple historical processing tasks; obtaining a reference energy consumption and a reference duration of each reference processing task, and predicting the energy consumption of the benchmark cluster to process the corresponding task to be processed based on the multiple reference energy consumptions, so as to obtain a benchmark energy consumption, and predicting the duration of the benchmark cluster to process the corresponding task to be processed based on the multiple reference durations, so as to obtain a benchmark duration; a conversion module, configured to convert the energy consumption of each of the first clusters according to the reference energy consumption and the energy consumption proportional factor to obtain a corresponding first energy consumption, and to convert the duration of each of the first clusters according to the reference duration and the duration proportional factor to obtain a corresponding first duration; An input module, used for inputting the benchmark energy consumption and the benchmark duration corresponding to the benchmark cluster, and the first energy consumption and the first duration corresponding to each first cluster into a preset target model to obtain a model to be solved, wherein the model to be solved includes a comprehensive energy consumption index under the allocation relationship between multiple tasks to be processed and multiple intelligent computing clusters; A solution module is used to perform a minimization solution of the comprehensive energy consumption index based on the model to be solved, determine the allocation relationship between the multiple tasks to be processed and the multiple intelligent computing clusters, and allocate the multiple tasks to be processed to the multiple intelligent computing clusters according to the allocation relationship.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the resource scheduling method based on the intelligent computing cluster as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the resource scheduling method based on an intelligent computing cluster as described in any one of claims 1 to 6 is implemented.

Images