Task scheduling method and device based on multi-dimensional resources

Abstract

The application provides a kind of task scheduling method and device based on multidimensional resource, belong to cloud computing technical field, the task scheduling method based on multidimensional resource of the present application, by collecting multidimensional resource state data including energy index and resource demand prediction value, construct multi-objective optimization function, and dynamically adjust the weight coefficient of each target in the multi-objective optimization function according to the load state of the current system, and generate optimal scheduling strategy, realize the perception of multidimensional resource data and the optimization of energy efficiency, can guarantee performance and balance energy consumption, so that it can significantly reduce power consumption under the premise of guaranteeing business service level, realize the double optimization of performance and energy consumption.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a task scheduling method and apparatus based on multi-dimensional resources. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, data centers are growing in scale, and the types and complexity of computing tasks are increasing exponentially. In existing distributed computing systems, resource scheduling involves the rational allocation of tasks to be processed across physical or virtual computing nodes.

[0003] Currently, mainstream resource scheduling technologies, such as the Kubernetes default scheduler, typically employ rule-based static allocation strategies. These strategies might match the resource requests declared by tasks with the remaining resources on nodes, or use algorithms like round-robin or shortest job first. However, these existing scheduling schemes have significant drawbacks. They lack the ability to perceive multi-dimensional resources, and static strategies cannot adapt to dynamically changing workloads. They often only respond passively after nodes are overloaded or service quality is compromised, leading to increased service latency.

[0004] Therefore, there is an urgent need for a scheduling scheme that can perceive the status of resources in multiple dimensions in real time and balance performance and energy consumption. Summary of the Invention

[0005] This invention provides a task scheduling method and apparatus based on multi-dimensional resources, which solves the shortcomings of existing technologies that cannot fully perceive resource data to achieve high-performance and energy-saving scheduling, and realizes scheduling that can perceive the status of multi-dimensional resources and balance performance and energy consumption.

[0006] This invention provides a task scheduling method based on multi-dimensional resources, which is applied to a distributed computing system. The method includes: Real-time acquisition of resource status data of each physical node in the computing cluster, the resource status data including at least computing resource indicators, storage resource indicators, network resource indicators and energy indicators; Obtain historical operation data and task feature descriptions of the task to be scheduled, and use a pre-trained hybrid prediction model to predict the resource demand of the task to be scheduled in the future time period to obtain the predicted resource demand value. Based on the resource status data and the predicted resource demand, a multi-objective optimization function is constructed, which includes system performance, energy efficiency, and resource cost. The weight coefficients of each objective in the multi-objective optimization function are dynamically adjusted according to the current system load status, and an optimal scheduling strategy is generated. According to the optimal scheduling strategy, the task to be scheduled is distributed to the target physical node for execution, or the running task is migrated from the current node to the target physical node.

[0007] According to the present invention, a task scheduling method based on multi-dimensional resources is provided, wherein the real-time acquisition of resource status data of each physical node in the computing cluster includes: encoding the resource status data into a unified resource status vector; the computing resource indicators include CPU utilization, GPU load, and memory bandwidth; the storage resource indicators include input / output operations per second (IOPS) and throughput; the network resource indicators include bandwidth utilization and latency; and the energy indicators include real-time power consumption and temperature of the nodes.

[0008] According to the present invention, a task scheduling method based on multi-dimensional resources is provided, wherein the hybrid prediction model includes a first prediction sub-model and a second prediction sub-model configured in parallel; the step of using the pre-trained hybrid prediction model to predict the resource demand of the task to be scheduled in a future time period to obtain a predicted resource demand value includes: The historical operational data is processed using the first prediction sub-model, and the short-term resource demand forecast is calculated using the differential integrated moving average autoregressive algorithm. The task feature description is processed using the second prediction sub-model, and the long-term resource demand trend value is calculated using the long short-term memory network algorithm. Based on a preset fusion weight, the short-term resource demand forecast and the long-term resource demand trend are weighted and summed to obtain the final resource demand forecast.

[0009] According to the present invention, a task scheduling method based on multi-dimensional resources, wherein dynamically adjusting the weight coefficients of each objective in the multi-objective optimization function according to the current system load state includes: Obtain the current total load value of the distributed computing system; If the current total load value is higher than a preset first load threshold, the weighting coefficient corresponding to system performance is increased and the weighting coefficient corresponding to energy efficiency is decreased; if the current total load value is lower than a preset second load threshold, the weighting coefficient corresponding to system performance is decreased and the weighting coefficient corresponding to energy efficiency is increased; if the current total load value is between the first load threshold and the second load threshold, the weighting coefficients corresponding to system performance and energy efficiency are both set to a balanced value; the first load threshold is greater than the second load threshold.

[0010] According to the present invention, a task scheduling method based on multi-dimensional resources is provided, wherein the generation of the optimal scheduling strategy includes: A population with multiple scheduling schemes is initialized based on a non-dominated sorting genetic algorithm. The fitness of each individual in the population is calculated based on the predicted resource demand and the multi-objective optimization function adjusted by the weight coefficients. The population is iteratively evolved through selection, crossover, and mutation operators until a preset termination condition is met, generating the optimal solution set. The solution with the highest fitness is selected from the set of optimal solutions as the optimal scheduling strategy.

[0011] According to the task scheduling method based on multi-dimensional resources provided by the present invention, before constructing the multi-objective optimization function including system performance, energy efficiency and resource cost, the method further includes: normalizing the data of each dimension in the resource state vector.

[0012] According to the present invention, a task scheduling method based on multi-dimensional resources is provided, the method further includes: Periodically calculate the utilization deviation between the actual resource utilization rate of the currently running task and the predicted resource demand; Determine the probability of the current operational task violating service level objectives; When the utilization deviation exceeds a preset deviation threshold, or the risk probability exceeds a preset risk threshold, the weight coefficients of each objective in the multi-objective optimization function are dynamically adjusted according to the current system load status to regenerate the scheduling strategy.

[0013] According to a task scheduling method based on multi-dimensional resources provided by the present invention, determining the risk probability of a currently running task violating service level objectives includes: Monitor the response time and throughput of currently running tasks; Based on the historical performance fluctuation characteristics of the task, calculate the conditional probability that the response time exceeds the service level target limit under the current resource configuration.

[0014] The present invention also provides a task scheduling device based on multi-dimensional resources, comprising: The acquisition module is used to collect resource status data of each physical node in the computing cluster in real time. The resource status data includes at least computing resource indicators, storage resource indicators, network resource indicators, and energy indicators. The acquisition module is used to acquire historical running data and task feature descriptions of the task to be scheduled, and to use a pre-trained hybrid prediction model to predict the resource demand of the task to be scheduled in the future time period to obtain the predicted value of resource demand. The first processing module is used to construct a multi-objective optimization function that includes system performance, energy efficiency and resource cost based on the resource status data and the resource demand forecast; dynamically adjust the weight coefficients of each objective in the multi-objective optimization function according to the current system load status, and generate the optimal scheduling strategy; The second processing module is used to control the task to be scheduled to be distributed to the target physical node for execution according to the optimal scheduling strategy, or to migrate the running task from the current node to the target physical node.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the task scheduling method based on multi-dimensional resources as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the task scheduling method based on multi-dimensional resources as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the task scheduling method based on multi-dimensional resources as described above.

[0018] The task scheduling method and apparatus based on multi-dimensional resources provided by this invention collects multi-dimensional resource status data, including energy indicators, and resource demand prediction values, constructs a multi-objective optimization function, and dynamically adjusts the weight coefficients of each objective in the multi-objective optimization function according to the current system load status, and generates an optimal scheduling strategy. This achieves perception of multi-dimensional resource data and optimization of energy efficiency, ensuring performance while balancing energy consumption. It enables a significant reduction in power consumption while ensuring the level of business service, achieving dual optimization of performance and energy consumption. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the task scheduling method based on multi-dimensional resources provided by the present invention; Figure 2 is a schematic diagram of the task scheduling device based on multi-dimensional resources provided by the present invention; Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] The following is combined with Figures 1-3 This invention describes a task scheduling method and apparatus based on multi-dimensional resources.

[0023] This embodiment provides a task scheduling method based on multi-dimensional resources. This method can be applied to distributed computing systems, which may include, but are not limited to, cloud computing data centers, edge computing node clusters, or 5G network slice resource pools. See also Figure 1 The method mainly includes steps 101, 102, 103 and 104.

[0024] Step 101: Collect resource status data of each physical node in the computing cluster in real time.

[0025] It's important to note that the physical status of each physical node in the computing cluster can be obtained through monitoring agents deployed at the kernel layer, etc. The computing cluster can be a heterogeneous server cluster built on the Kubernetes container orchestration platform. A physical node refers to an electronic device with independent computing, storage, and network communication capabilities. In distributed systems such as Kubernetes cluster architectures, physical nodes can be host machines, worker nodes, or servers.

[0026] In existing scheduling systems such as Kubernetes, resource awareness is often limited to the static allocation of CPU and memory. However, in this implementation, by deploying a high-performance monitoring agent on each physical node or utilizing eBPF (Extended Berkeley Packet Filter) technology, millisecond-level collection of underlying hardware metrics can be achieved.

[0027] The collected resource status data is multi-dimensional, including at least computing resource indicators, storage resource indicators, network resource indicators, and energy indicators.

[0028] In some embodiments, real-time acquisition of resource status data of each physical node in the computing cluster includes: encoding the resource status data into a unified resource status vector; computing resource indicators include CPU utilization, GPU load and memory bandwidth; storage resource indicators include IOPS (Input / Output Operations Per Second) and throughput; network resource indicators include bandwidth utilization and latency; and energy indicators include real-time power consumption and temperature of the nodes.

[0029] In order to enable computer algorithms to uniformly process heterogeneous physical indicators, the collected raw data can be encoded into resource state vectors in a unified format.

[0030] Computing resource metrics include not only traditional CPU utilization, but also GPU load and memory bandwidth. In high-concurrency scenarios such as deep learning training, the CPU is often not the bottleneck; saturation of GPU memory bandwidth or memory bus bandwidth can lead to performance degradation. Monitoring these metrics can effectively avoid performance interference caused by bandwidth contention.

[0031] Storage resource metrics include read / write operations per second and throughput. When scheduling tasks, it's possible to differentiate between compute-intensive and I / O-intensive tasks, avoiding scheduling multiple high-I / O tasks to the same node and preventing disk queue congestion.

[0032] Network resource metrics include bandwidth utilization and latency. By monitoring link latency, frequently communicating microservices can be scheduled to nodes closer in the network topology, such as those in the same rack or under the same switch, thereby reducing communication overhead.

[0033] Energy metrics include real-time node power consumption and temperature. The real-time power consumption of a node also reflects the task execution status; different power consumption levels result in different hardware utilization rates. When the CPU temperature of a physical node is too high, even if its utilization rate has not reached 100%, new tasks should not be scheduled for it to prevent forced frequency reduction due to hardware thermal protection, thereby ensuring the stability of task operation.

[0034] Understandably, by constructing this full-dimensional resource state vector, the method in this embodiment, which senses GPU, memory bandwidth, and temperature, makes the system suitable for heterogeneous computing environments that mix complex workloads such as AI training and big data processing, effectively improving the overall utilization of hardware resources and system stability.

[0035] Before constructing a multi-objective optimization function that includes system performance, energy efficiency, and resource costs, the method also includes normalizing the data of each dimension in the resource state vector.

[0036] Because physical metrics of different dimensions have vastly different units—for example, network bandwidth is at the Gbps level and can reach 10,000; while CPU utilization is a percentage and can range from 0 to 1; and temperature is in the tens of degrees Celsius—direct calculation can cause features with large values ​​to mask features with small values.

[0037] This embodiment can use the max-min normalization or Z-Score normalization method to map all dimensions of data to the [0,1] interval or a standard normal distribution.

[0038] For example, the max-min normalization method can be used, and the conversion formula is as follows: ; Among them, X norm X is the value after conversion, and X is the value before conversion. max X is the largest value in the sample. min It is the smallest value in the sample.

[0039] For some indicators with long-tail distribution characteristics, such as network latency, the Z-Score normalization method can be optionally used to map them to a distribution with a mean of 0 and a variance of 1.

[0040] Understandably, normalization can eliminate the influence of dimensions, allowing the weight coefficients in the optimization function to truly reflect the importance of each objective. It also significantly accelerates the convergence speed of the model and improves computational efficiency.

[0041] Step 102: Obtain historical operation data and task feature descriptions of the task to be scheduled, and use a pre-trained hybrid prediction model to predict the resource demand of the task to be scheduled in the future time period to obtain the predicted value of resource demand.

[0042] This embodiment can acquire historical execution logs of tasks to be scheduled, such as microservice containers and big data batch processing jobs, as well as static feature descriptions of the tasks, such as image size and startup command parameters. This data can be input into a pre-trained hybrid prediction model. Based on past behavior patterns, the hybrid prediction model can infer the resource demand change curve of the task over a future time period, thereby obtaining accurate resource demand predictions.

[0043] Unlike existing technologies that rely solely on static request values ​​submitted by users, this step obtains historical operational data of the task to be scheduled, such as time-series logs and task feature descriptions, and inputs this data into a pre-trained hybrid prediction model. The hybrid prediction model utilizes statistical and deep learning methods to output a predicted resource demand for the task in a future time period, such as the next 10 minutes or 1 hour. This predicted value is not merely a scalar value, but can be a demand curve vector that varies over time.

[0044] Step 103: Based on resource status data and resource demand forecasts, construct a multi-objective optimization function that includes system performance, energy efficiency, and resource costs; dynamically adjust the weight coefficients of each objective in the multi-objective optimization function according to the current system load status, and generate the optimal scheduling strategy.

[0045] It should be noted that a multi-objective optimization function can be constructed, encompassing system performance, energy efficiency, and resource costs. This multi-objective optimization function is not static but a dynamic equation with load-aware capabilities. It can assess the current load level of the entire cluster in real time and dynamically adjust the weight coefficients of each sub-objective in the optimization function accordingly. Sub-objectives can include performance, energy efficiency, cost, etc.

[0046] Specifically, a multi-objective optimization function can be constructed based on collected supply-side data, such as resource status, and predicted demand-side data, such as resource demand. Instead of using fixed weighting coefficients, the function can dynamically tilt towards performance or energy efficiency based on the current total load level of the cluster, thereby generating the optimal scheduling strategy. For example, it can automatically tilt towards performance during peak business periods and towards energy efficiency during off-peak periods. Based on the adjusted function, an optimal scheduling strategy that balances the interests of all parties can be solved.

[0047] Step 104: Based on the optimal scheduling strategy, control the task to be scheduled to be distributed to the target physical node for execution, or migrate the running task from the current node to the target physical node.

[0048] Based on the strategy, specific instructions can be generated to control the distribution of tasks to selected target physical nodes through the container's interface, or a real-time migration process can be triggered to seamlessly transfer running tasks. According to the calculated optimal strategy, the scheduler sends instructions to the cluster management interface such as Kube-apiserver to bind the tasks to be scheduled to the specified target physical node; or in the case of rescheduling, a hot migration process can be triggered to migrate running containers or virtual machines to a new node.

[0049] The task scheduling method based on multi-dimensional resources provided in this embodiment of the invention collects multi-dimensional resource status data, including energy indicators, and resource demand prediction values, constructs a multi-objective optimization function, dynamically adjusts the weight coefficients of each objective in the multi-objective optimization function according to the current system load status, and generates an optimal scheduling strategy. This achieves perception of multi-dimensional resource data and optimization of energy efficiency, ensuring performance while balancing energy consumption. It enables a significant reduction in power consumption while ensuring the level of business services, achieving dual optimization of performance and energy consumption.

[0050] In some embodiments, the hybrid prediction model includes a first prediction sub-model and a second prediction sub-model configured in parallel. The pre-trained hybrid prediction model is used to predict the resource demand of the task to be scheduled within a future time period to obtain a predicted resource demand value. This includes: processing historical operational data using the first prediction sub-model and calculating a short-term resource demand prediction value using a differential integrated moving average autoregressive algorithm; processing task feature descriptions using the second prediction sub-model and calculating a long-term resource demand trend value using a long short-term memory network algorithm; and weighting and summing the short-term resource demand prediction value and the long-term resource demand trend value based on preset fusion weights to obtain the final predicted resource demand value.

[0051] The first prediction sub-model is used to process short-term historical data with periodic and stationary characteristics. The first prediction sub-model can employ the Autoregressive Integrated Moving Average (ARIMA) algorithm.

[0052] The first predictive sub-model takes as input a sequence of task resource usage over the past T time steps, such as a CPU utilization sequence. It first performs d-order differencing on the sequence to eliminate non-stationarity, then incorporates an autoregressive term to capture the influence of historical values ​​and a moving average term to capture the influence of prediction errors. The output of the first predictive sub-model can be a short-term resource demand forecast. It is understandable that the ARIMA algorithm can capture linear relationships and short-term fluctuations, and has extremely high computational efficiency and accuracy, requiring no large amount of training data for cold start.

[0053] The second prediction sub-model is used to handle long-term, non-linear task feature data with complex dependencies. The second prediction sub-model can employ a Long Short-Term Memory (LSTM) algorithm. The input to the second prediction sub-model can be a tensor dimension of the input data. The feature dimension includes the historical load sequence of the task, the task type, and submission time features such as whether it is a weekday and the specific hour.

[0054] The LSTM network architecture consists of an input layer, an LSTM layer, and a fully connected layer. The input layer receives the preprocessed multidimensional feature vector. The LSTM layer contains multiple stacked LSTM units. Each unit contains a forget gate, an input gate, and an output gate. The forget gate determines the information to discard from the unit state; the input gate determines the information to update the unit state; and the unit state update is used to maintain long-term memory. The fully connected layer maps the output of the LSTM layer to the final predicted value. The output layer outputs the long-term resource demand trend value.

[0055] Understandably, LSTM can solve the vanishing gradient problem in traditional recurrent neural networks through gating mechanisms, and can capture the evolution of task resource requirements over a long period of time. For example, a certain type of task always reaches its peak 10 minutes after startup.

[0056] After obtaining the short-term resource demand forecast and the long-term resource demand trend, a weighted sum can be calculated through fusion computation. In some embodiments, the weights can be preset manually or dynamically learned through an attention mechanism; in other embodiments, the weights can also be hyperparameters that are adaptively adjusted based on historical statistical values ​​of the prediction error.

[0057] In some embodiments, millions of task execution trajectories can be extracted from historical logs of a distributed system, such as Prometheus monitoring data. The data is then divided into training, validation, and test sets in an 8:1:1 ratio. Specifically, Mean Squared Error (MSE) can be used as the loss function, with the loss function formula as follows: ; Where N is the number of training rounds, y true y represents the actual resource usage of the task. pred This is the model's predicted output.

[0058] In some embodiments, the Adam (Adaptive Moment Estimation) optimizer can be used for parameter updates, with an initial learning rate set to 0.001, along with a learning rate decay strategy. When the loss on the validation set no longer decreases within five consecutive training epochs, an early stopping mechanism is triggered, ending training. Finally, the model is validated and tested using both validation and test sets.

[0059] The first predictive sub-model processes the short-term, periodic components of historical operational data. It employs the Differential Integrated Moving Average Autoregressive (ARIMA) algorithm, which can quickly capture short-term fluctuations in task resource usage, such as heartbeat fluctuations every 5 minutes.

[0060] The second predictive sub-model handles task feature descriptions and long-term dependencies, employing a Long Short-Term Memory (LSTM) network. Since cloud task load variations are often non-linear and influenced by various external factors such as promotional activities and holidays, traditional linear models struggle to fit these variations. LSTM, through its internal forget gate, input gate, and output gate mechanisms, effectively preserves crucial information in long-sequence data, capturing long-term trends in resource demand over time.

[0061] Based on this, the short-term resource demand forecasts output by ARIMA and the long-term resource demand trend values ​​output by LSTM are weighted and summed according to the preset fusion weights.

[0062] Understandably, combining the short-term sensitivity of the ARIMA algorithm with the long-term memory capability of the LSTM model can solve the problem of low prediction accuracy of a single model when facing complex and ever-changing cloud computing loads. Using ARIMA alone cannot handle complex nonlinear trends, while using LSTM alone faces difficulties in cold start and high computational overhead. This embodiment, by combining short-term and long-term sensitivity, and complementing linear and nonlinear approaches, significantly reduces inference latency while ensuring prediction accuracy. It can accurately predict resource peaks for tasks in the future, providing data support for advance capacity expansion or off-peak scheduling.

[0063] In some embodiments, dynamically adjusting the weight coefficients of each objective in the multi-objective optimization function according to the current system load status includes: obtaining the current total load value of the distributed computing system; if the current total load value is higher than a preset first load threshold, increasing the weight coefficient corresponding to system performance and decreasing the weight coefficient corresponding to energy efficiency; if the current total load value is lower than a preset second load threshold, decreasing the weight coefficient corresponding to system performance and increasing the weight coefficient corresponding to energy efficiency; if the current total load value is between the first load threshold and the second load threshold, setting the weight coefficients corresponding to system performance and energy efficiency to a balanced value; the first load threshold is greater than the second load threshold.

[0064] It allows for real-time monitoring of the current total load of a distributed system. For example, this can be represented by the cluster's average CPU utilization or the number of active tasks.

[0065] In high-load mode, where the load exceeds a first load threshold (e.g., 80%), the system is on the verge of congestion. The system performance weight can be automatically increased while the energy efficiency weight is decreased. The scheduling strategy will favor load balancing, distributing tasks across any node with available resources, which may increase energy consumption to prevent node downtime.

[0066] In low-load mode, where the load is below the second threshold (e.g., 30%), a large amount of system resources are idle. Performance weighting can be reduced, while energy efficiency weighting can be significantly increased. Scheduling strategies can concentrate tasks on a few high-performance servers, allowing the remaining idle servers to enter deep sleep mode to save power.

[0067] In balanced mode, where the load is less than or equal to the first load threshold and greater than or equal to the second threshold, the current total load value is between the first and second load thresholds. The weight can be set to a balanced value to take into account both response speed and operating costs.

[0068] Specifically, performance goals can minimize task completion time or response latency, which can be achieved by maximizing resource matching. Energy efficiency goals can minimize the total power consumption of the cluster, which can be achieved by consolidating tasks onto fewer active nodes, putting some nodes into a dormant state, or scheduling tasks to nodes with higher energy efficiency. Cost goals can minimize the total amount of paid resources used.

[0069] Understandably, existing scheduling algorithms typically use fixed weights, resulting in all servers running at full speed during low-load periods at night, leading to significant energy waste; or limiting resources during peak periods to save energy, causing service crashes. This embodiment, through a weight adjustment mechanism, enables the data center to have adjustment capabilities, maximizing overall efficiency in 24 / 7 operation scenarios.

[0070] In some embodiments, generating the optimal scheduling strategy includes: initializing a population containing multiple scheduling schemes based on a non-dominated sorting genetic algorithm; calculating the fitness of each individual in the population according to the predicted resource demand and the multi-objective optimization function adjusted by the weight coefficients; iteratively evolving the population through selection, crossover and mutation operators until a preset termination condition is met to generate an optimal solution set; and selecting the solution with the highest fitness from the optimal solution set as the optimal scheduling strategy.

[0071] Understandably, this embodiment employs an improved non-dominated sorting genetic algorithm. First, the population is initialized, and a set of scheduling schemes is randomly generated, with each gene representing a mapping between a task and a physical node. The obtained resource demand predictions and dynamic weights are substituted into a multi-objective optimization function to calculate the comprehensive score, or fitness, of each scheme, and schemes with high fitness are retained.

[0072] Crossover and mutation simulate biological evolution by exchanging task allocation fragments in different schemes or randomly changing the target node of a task to explore the solution space and prevent getting trapped in local optima. A Pareto Front is selected through non-dominated sorting, meaning that no other solution in the set is superior to it on all objectives. Finally, after several generations of evolutionary convergence, the scheme with the highest fitness is selected from the Pareto-optimal solution set as the final scheduling strategy.

[0073] This embodiment utilizes the global search capability of genetic algorithms to find near-optimal solutions among a massive number of possible scheduling combinations in a very short time. This avoids the short-sightedness of algorithms such as greedy algorithms and can generate fine-grained scheduling instructions that satisfy performance constraints and are as energy-efficient as possible on a global scale.

[0074] In some embodiments, the task scheduling method based on multi-dimensional resources of the present invention further includes: periodically calculating the utilization deviation between the actual resource utilization rate of the currently running task and the predicted value of resource demand; determining the risk probability of the currently running task violating the service level target; and when the utilization deviation exceeds a preset deviation threshold or the risk probability exceeds a preset risk threshold, dynamically adjusting the weight coefficients of each objective in the multi-objective optimization function according to the current system load status to regenerate the scheduling strategy.

[0075] It can periodically calculate the difference between the current actual resource utilization rate of a task and the predicted resource demand before scheduling. If the deviation exceeds a threshold such as 20%, it indicates that the prediction is inaccurate or there is a sudden change in business needs, and correction is required.

[0076] It can monitor the response time and throughput of running tasks and calculate conditional probabilities based on historical fluctuation characteristics. That is, in the current congested resource environment, the probability that the response time of a task will exceed the service level target limit within a short period of time in the future. If the probability exceeds a preset risk threshold, such as a 5% default risk, a preventative rescheduling can be triggered even if there is no current default.

[0077] Determine the probability of the current running task violating the service level target, including: monitoring the response time and throughput of the current running task; and calculating the conditional probability that the response time exceeds the service level target limit under the current resource configuration based on the historical performance fluctuation characteristics of the task.

[0078] For example, a probability density function for response time, such as a Gaussian distribution, can be constructed based on the historical performance data of the task. The integral value of the function curve in the region to the right of the service level target threshold under the current resource conditions, i.e., the tail cumulative probability, can then be calculated to obtain the conditional probability.

[0079] In this embodiment, in order to balance the overhead and benefits of rescheduling, dual triggering conditions can be set.

[0080] Condition 1: Utilization Deviation Trigger. The actual resource utilization of a task can be periodically compared with previous forecasts. A large deviation indicates inaccurate forecasting or a sudden change in load, requiring replanning.

[0081] Condition 2: Service level target risk is triggered. Even if utilization appears normal, intervention is required if service quality declines.

[0082] For condition two, this embodiment can use a probabilistic model for evaluation. The response time and throughput of running tasks can be monitored in real time. Based on the historical performance model of this task type, the probability that the response time exceeds the service level target limit under the current resource configuration, such as CPU limits and memory bandwidth, is calculated. If the probability is greater than a preset risk threshold, such as 5%, the task is considered to be in a dangerous state and requires immediate rescheduling to migrate it to a node with more abundant resources or less interference.

[0083] The task scheduling device based on multi-dimensional resources provided by the present invention is described below. The task scheduling device based on multi-dimensional resources described below and the task scheduling method based on multi-dimensional resources described above can be referred to in correspondence.

[0084] like Figure 2 As shown, the task scheduling device based on multi-dimensional resources in this embodiment of the invention mainly includes a collection module 210, an acquisition module 220, a first processing module 230, and a second processing module 240.

[0085] The acquisition module 210 is used to collect resource status data of each physical node in the computing cluster in real time. The resource status data includes at least computing resource indicators, storage resource indicators, network resource indicators and energy indicators. The acquisition module 220 is used to acquire historical running data and task feature descriptions of the task to be scheduled, and to use a pre-trained hybrid prediction model to predict the resource demand of the task to be scheduled in the future time period to obtain the predicted value of resource demand. The first processing module 230 is used to construct a multi-objective optimization function that includes system performance, energy efficiency and resource cost based on resource status data and resource demand forecasts; dynamically adjust the weight coefficients of each objective in the multi-objective optimization function according to the current system load status, and generate the optimal scheduling strategy. The second processing module 240 is used to control the distribution of tasks to be scheduled to the target physical node for execution according to the optimal scheduling strategy, or to migrate running tasks from the current node to the target physical node.

[0086] The task scheduling device based on multi-dimensional resources provided in this embodiment of the invention collects multi-dimensional resource status data, including energy indicators, and resource demand prediction values, constructs a multi-objective optimization function, dynamically adjusts the weight coefficients of each objective in the multi-objective optimization function according to the current system load status, and generates an optimal scheduling strategy. This realizes the perception of multi-dimensional resource data and the optimization of energy efficiency, ensuring performance while balancing energy consumption. It enables a significant reduction in power consumption while ensuring the level of business service, achieving dual optimization of performance and energy consumption.

[0087] Figure 3An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a task scheduling method based on multi-dimensional resources. This method includes: real-time acquisition of resource status data of each physical node in the computing cluster, the resource status data including at least computing resource indicators, storage resource indicators, network resource indicators, and energy indicators; acquisition of historical running data and task feature descriptions of the tasks to be scheduled, using a pre-trained hybrid prediction model to predict the resource demand of the tasks to be scheduled in the future time period, and obtaining the predicted resource demand value; based on the resource status data and the predicted resource demand value, constructing a multi-objective optimization function including system performance, energy efficiency, and resource cost; dynamically adjusting the weight coefficients of each objective in the multi-objective optimization function according to the current system load status, and generating an optimal scheduling strategy; according to the optimal scheduling strategy, controlling the tasks to be scheduled to be distributed to target physical nodes for execution, or migrating running tasks from the current node to the target physical node.

[0088] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the task scheduling method based on multi-dimensional resources provided by the above methods. The method includes: real-time acquisition of resource status data of each physical node in the computing cluster, the resource status data including at least computing resource indicators, storage resource indicators, network resource indicators, and energy indicators; acquisition of historical running data and task feature descriptions of the task to be scheduled, prediction of the resource demand of the task to be scheduled in a future time period using a pre-trained hybrid prediction model, and obtaining a predicted value of resource demand; construction of a multi-objective optimization function including system performance, energy efficiency, and resource cost based on the resource status data and the predicted value of resource demand; dynamic adjustment of the weight coefficients of each objective in the multi-objective optimization function according to the current system load status, and generation of an optimal scheduling strategy; and control of the task to be scheduled to be distributed to the target physical node for execution, or migration of the running task from the current node to the target physical node according to the optimal scheduling strategy.

[0090] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the task scheduling method based on multi-dimensional resources provided by the above methods. The method includes: real-time acquisition of resource status data of each physical node in the computing cluster, the resource status data including at least computing resource indicators, storage resource indicators, network resource indicators, and energy indicators; acquisition of historical running data and task feature descriptions of the task to be scheduled, and prediction of the resource demand of the task to be scheduled in a future time period using a pre-trained hybrid prediction model to obtain a predicted value of resource demand; construction of a multi-objective optimization function including system performance, energy efficiency, and resource cost based on the resource status data and the predicted value of resource demand; dynamic adjustment of the weight coefficients of each objective in the multi-objective optimization function according to the current system load status, and generation of an optimal scheduling strategy; and control the task to be scheduled to be distributed to the target physical node for execution, or the running task to be migrated from the current node to the target physical node according to the optimal scheduling strategy.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A task scheduling method based on multi-dimensional resources, characterized in that, The method is applied to a distributed computing system, and the method includes: Real-time acquisition of resource status data of each physical node in the computing cluster, the resource status data including at least computing resource indicators, storage resource indicators, network resource indicators and energy indicators; Obtain historical operation data and task feature descriptions of the task to be scheduled, and use a pre-trained hybrid prediction model to predict the resource demand of the task to be scheduled in the future time period to obtain the predicted resource demand value. Based on the resource status data and the predicted resource demand, a multi-objective optimization function is constructed, which includes system performance, energy efficiency, and resource cost. The weight coefficients of each objective in the multi-objective optimization function are dynamically adjusted according to the current system load status, and an optimal scheduling strategy is generated. According to the optimal scheduling strategy, the task to be scheduled is distributed to the target physical node for execution, or the running task is migrated from the current node to the target physical node.

2. The task scheduling method based on multi-dimensional resources according to claim 1, characterized in that, The real-time acquisition of resource status data of each physical node in the computing cluster includes: encoding the resource status data into a unified resource status vector; the computing resource indicators include CPU utilization, GPU load, and memory bandwidth; the storage resource indicators include input / output operations per second (IOPS) and throughput; the network resource indicators include bandwidth utilization and latency; and the energy indicators include real-time power consumption and temperature of the nodes.

3. The task scheduling method based on multi-dimensional resources according to claim 1, characterized in that, The hybrid prediction model includes a first prediction sub-model and a second prediction sub-model configured in parallel; the process of using the pre-trained hybrid prediction model to predict the resource demand of the task to be scheduled within a future time period to obtain the predicted resource demand value includes: The historical operational data is processed using the first prediction sub-model, and the short-term resource demand forecast is calculated using the differential integrated moving average autoregressive algorithm. The task feature description is processed using the second prediction sub-model, and the long-term resource demand trend value is calculated using the long short-term memory network algorithm. Based on a preset fusion weight, the short-term resource demand forecast and the long-term resource demand trend are weighted and summed to obtain the final resource demand forecast.

4. The task scheduling method based on multi-dimensional resources according to claim 1, characterized in that, The step of dynamically adjusting the weight coefficients of each objective in the multi-objective optimization function according to the current system load status includes: Obtain the current total load value of the distributed computing system; If the current total load value is higher than a preset first load threshold, the weighting coefficient corresponding to system performance is increased and the weighting coefficient corresponding to energy efficiency is decreased; if the current total load value is lower than a preset second load threshold, the weighting coefficient corresponding to system performance is decreased and the weighting coefficient corresponding to energy efficiency is increased; if the current total load value is between the first load threshold and the second load threshold, the weighting coefficients corresponding to system performance and energy efficiency are both set to a balanced value; the first load threshold is greater than the second load threshold.

5. The task scheduling method based on multi-dimensional resources according to claim 1, characterized in that, The optimal scheduling strategy includes: A population with multiple scheduling schemes is initialized based on a non-dominated sorting genetic algorithm. The fitness of each individual in the population is calculated based on the predicted resource demand and the multi-objective optimization function adjusted by the weight coefficients. The population is iteratively evolved through selection, crossover, and mutation operators until a preset termination condition is met, generating the optimal solution set. The solution with the highest fitness is selected from the set of optimal solutions as the optimal scheduling strategy.

6. The task scheduling method based on multi-dimensional resources according to claim 2, characterized in that, Before constructing the multi-objective optimization function that includes system performance, energy efficiency, and resource cost, the method further includes: normalizing the data of each dimension in the resource state vector.

7. The task scheduling method based on multi-dimensional resources according to claim 1, characterized in that, The method further includes: Periodically calculate the utilization deviation between the actual resource utilization rate of the currently running task and the predicted resource demand; Determine the probability of the current operational task violating service level objectives; When the utilization deviation exceeds a preset deviation threshold, or the risk probability exceeds a preset risk threshold, the weight coefficients of each objective in the multi-objective optimization function are dynamically adjusted according to the current system load status to regenerate the scheduling strategy.

8. The task scheduling method based on multi-dimensional resources according to claim 7, characterized in that, Determining the probability of the current running task violating the service level target includes: Monitor the response time and throughput of currently running tasks; Based on the historical performance fluctuation characteristics of the task, calculate the conditional probability that the response time exceeds the service level target limit under the current resource configuration.

9. A task scheduling device based on multi-dimensional resources, characterized in that, include: The acquisition module is used to collect resource status data of each physical node in the computing cluster in real time. The resource status data includes at least computing resource indicators, storage resource indicators, network resource indicators, and energy indicators. The acquisition module is used to acquire historical running data and task feature descriptions of the task to be scheduled, and to use a pre-trained hybrid prediction model to predict the resource demand of the task to be scheduled in the future time period to obtain the predicted value of resource demand. The first processing module is used to construct a multi-objective optimization function that includes system performance, energy efficiency and resource cost based on the resource status data and the resource demand forecast; dynamically adjust the weight coefficients of each objective in the multi-objective optimization function according to the current system load status, and generate the optimal scheduling strategy; The second processing module is used to control the task to be scheduled to be distributed to the target physical node for execution according to the optimal scheduling strategy, or to migrate the running task from the current node to the target physical node.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it implements the task scheduling method based on multi-dimensional resources as described in any one of claims 1 to 8.

Images