Intelligent scheduling method and system for Huaan Atlas heterogeneous computing resources based on dynamic load awareness

Abstract

The invention belongs to the technical field of computing resource scheduling, and particularly relates to an intelligent scheduling method and system for Huaan Atlas heterogeneous computing resources based on dynamic load awareness. The method comprises the steps that the real-time state of multi-dimensional hardware data is collected, and a basic data source is provided for subsequent steps; dynamically adapting tasks and hardware characteristics through a matching degree matrix, modeling aiming at various basic data, and constructing a state vector required by reinforcement learning; predicting a fault risk score through a lightweight prediction model deployed at each computing node; and a deep Q network is adopted as a model architecture, a state vector and a fault risk score are input, reinforcement learning training is performed through a reward function in a multi-target vector form, a final scheduling model is obtained, and a task allocation decision is output. The problems that in the prior art, the hardware state cannot be sensed in real time, hardware characteristic matching is ignored, consequently, the computing resource utilization rate is insufficient, and fault recovery is passive are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computing resource scheduling, and more particularly to a Huawei Atlas heterogeneous computing resource intelligent scheduling method and system based on dynamic load sensing. BACKGROUND

[0002] In the large-scale distributed training scene of Ascend AI processors and NPU / GPU hybrid clusters under the CANN architecture of the Huawei Atlas platform, hardware heterogeneity easily leads to load imbalance problems. There are mainly two kinds of current mainstream technical solutions: One is a static scheduling algorithm. This algorithm uses polling or priority queue to allocate tasks, and binds tasks to specific hardware such as Ascend NPU or NVIDIA GPU through preset rules. However, this algorithm has poor dynamic adaptability and cannot cope with task load fluctuations, such as real-time video processing and offline training mixed scenes, resulting in low resource utilization. Moreover, it does not optimize task allocation in combination with the Cube / Vector / Scalar unit characteristics of Ascend AI Core, resulting in high idle rate of computing units.

[0003] The second is the NSLB (Network Scale Load Balancing) load balancing algorithm. This algorithm optimizes task allocation by statistical historical communication data to improve the communication throughput of large-scale clusters. However, it has the following limitations in dynamic load scenarios: high feedback delay, relies on historical communication data statistics for decision-making, lacks real-time state sensing capability of hardware (such as temperature, memory bandwidth, and computing unit utilization), for example, in a burst video processing task, it may cause task accumulation and delay of more than 300 ms; only optimizes communication throughput, ignores hardware characteristics matching, focuses on network path optimization such as ROCE lossless network, but does not consider the adaptability of tasks and heterogeneous hardware units such as Ascend Cube Core and NVIDIA Tensor Core, resulting in insufficient utilization of computing units; there is a multi-objective conflict, aiming to maximize communication throughput, but not balancing energy consumption and delay; fault recovery is passive, relying on task migration after fault triggering, lacking proactive prediction and pre-scheduling capability for hardware failure risks such as NPU overheating and frequency reduction. SUMMARY

[0004] The present application aims to overcome at least one of the above-mentioned defects of the prior art, and provides a Huawei Atlas heterogeneous computing resource intelligent scheduling method based on dynamic load sensing to solve the problems of the prior art that cannot real-time sense hardware state, hardware characteristics matching is ignored, resulting in insufficient utilization of computing resources, and passive fault recovery.

[0005] The present application also discloses a system loaded with a Huawei Atlas heterogeneous computing resource intelligent scheduling method based on dynamic load sensing.

[0006] The detailed technical solutions of the present application are as follows: The embodiment provides a Huawei Atlas heterogeneous computing resource intelligent scheduling method based on dynamic load sensing, which comprises the following steps: S1, multi-dimensional state real-time sensing is implemented, and real-time state collection is performed on hardware resource data, task characteristic data and energy consumption data, so as to provide basic data sources for subsequent steps; S2, the task and the hardware characteristics are dynamically adapted through a matching degree matrix, the various basic data are modeled, and a state vector required by reinforcement learning is constructed; S3, a hardware fault prediction mechanism is introduced, a light-weight prediction model LightGBM deployed on each computing node is input with a temperature sequence of a node in a fixed time in the past, a node memory bandwidth fluctuation rate, a node utilization fluctuation rate and a node sudden restart number, and outputs a fault risk score; S4, a light-weight deep Q network is used as a model architecture, the state vector and the fault risk score are input, reinforcement learning training is performed through a reward function in a multi-objective vector form, a final scheduling model is obtained, and a task allocation decision is output.

[0007] According to the present application, the hardware resource data comprises: Ascending NPU utilization rate, NVIDIA GPU utilization rate, memory, bandwidth and temperature; wherein, the NPU comprises Cube, Vector and Scalar units, and the GPU comprises Tensor Core, CUDACore and SM general unit; The task characteristic data comprises: model type, computing intensity and priority; The energy consumption data comprises: single card and whole machine power consumption.

[0008] According to the present application, the specific steps of S2 are as follows: S21, a task-hardware matching degree matrix is constructed for NPU and GPU hardware resource data, and the matching degree calculation formula is as follows: (1) In formula (1), represents the matching degree of a specific task and a hardware unit, if the proportion of the computing demand of the task to the hardware capacity is larger, it means that the proportion of the task to the hardware resource is higher, and the task is more suitable to be allocated to the hardware unit; represents the computing amount required for task execution, represents the maximum computing capacity of the hardware unit, represents the total bandwidth resource required in task execution, represents the maximum bandwidth of the hardware unit; and is an experience weight; S22, splicing all matching degrees to obtain a matching degree vector, and splicing the matching degree vector with other basic data to obtain a state vector required by reinforcement learning : (2) wherein temp represents temperature, mem represents memory, and utils represents the utilization rate of NPU and GPU, and the total number of dimensions is 6, power represents energy consumption, and the above is hardware resource data; task_type_embed represents a task-hardware matching degree vector, and the total number of dimensions is 6, task_priority represents task priority, task_flops represents calculation intensity, and task_band represents calculation bandwidth, and the above is task feature data.

[0009] According to the application, the calculation formula of the failure risk score in S3 is preferably: (3) In formula (3), , represents the failure risk score; represents a temperature sequence of the node in the past 3 minutes, represents the memory bandwidth fluctuation rate of the node, represents the utilization fluctuation rate of the node, represents the number of sudden restarts of the node, is an activation function, , , and are weight coefficients.

[0010] According to the application, the reward function in the form of a multi-target vector is preferably: (4) wherein is a delay, is a resource utilization rate, is a node energy consumption, is a punishment scheduling hotspot, and is 1 when the node temperature is higher than a preset threshold of 80℃, and is 0 otherwise.

[0011] According to the application, the reinforcement learning training process is as follows: a. Initialize the online Q network and the target Q network , and set the experience revisit buffer size to 10000; b. Every 5 seconds, collect a state vector input into the online Q network, and output a scheduling action The scheduling action refers to a task allocation decision, and the current task is allocated to which computing unit; (5) Wherein is a parameter in the greedy strategy, initially 1, gradually reduced with the number of training rounds; c. When the current node risk score is greater than the preset threshold 0.7, triggering the fault tolerance strategy and the fault rollback strategy, otherwise executing the scheduling action; The fault tolerance strategy refers to backing up the current node to other stable nodes at the same time; the fault rollback strategy refers to rescheduling the task if there is no feedback within a fixed time; d. After executing the scheduling action, the reward is obtained , update the network parameters: (6) (7) (8) Wherein, y is the target Q value, r is the reward, s is the state, a is the action, and are the parameters of the online Q network and the target Q network respectively, B is the training batch size, i is the training step number, is the learning rate.

[0012] In another aspect of the application, a Huawei Atlas heterogeneous computing resource intelligent scheduling system based on dynamic load perception is provided, the system comprising: A multi-dimensional state perception module: for real-time collection of hardware resource data, energy consumption data and task characteristic data, the hardware resource data including Ascend NPU utilization rate, NVIDIA GPU utilization rate, memory bandwidth, temperature; the energy consumption data including single card / whole machine power consumption; the task characteristic data including model type, calculation intensity, priority; A hardware characteristic modeling engine: for constructing a task-hardware matching degree matrix based on the task characteristic data and hardware resource data, splicing the matching degree matrix with the hardware resource data, energy consumption data and task characteristic data to obtain a state vector required for reinforcement learning; A fault prediction and fault tolerance perception scheduling module: for inputting the temperature sequence of the node in the past fixed time, the node memory bandwidth fluctuation rate, the node utilization fluctuation rate and the node burst restart number through the lightweight prediction model deployed on each computing node, outputting a fault risk score, and executing a corresponding fault tolerance strategy according to the fault risk score; The reinforcement learning and scheduling strategy module is used for adopting a light deep Q network as a model architecture, inputting the state vector and the fault risk score, training through a reward function in the form of a multi-target vector, obtaining a final scheduling model, and outputting a task allocation decision.

[0013] Compared with the prior art, the method has the advantages that: (1) The method integrates hardware resource data, task characteristic data and energy consumption data, realizes real-time perception of multi-dimensional states of the hybrid hardware cluster, and realizes dynamic load balancing.

[0014] (2) The method realizes accurate adaptation of tasks to Ascend AI Core and Nvidia GPU Core units through a matching degree matrix, and realizes millisecond-level scheduling through light DQN in combination with reinforcement learning and scheduling strategies, thereby supporting dynamic load fluctuation scenarios.

[0015] (3) The method actively predicts faults of the computing nodes, combines LightGBM prediction and risk assessment, changes passive fault recovery to active fault recovery, and improves the success rate of fault recovery. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of the Huawei Atlas heterogeneous computing resource intelligent scheduling method based on dynamic load perception according to the present application.

[0017] Figure 2 is a real-time multi-dimensional state perception schematic diagram according to embodiment 1 of the present application.

[0018] Figure 3 is a hardware characteristic modeling engine schematic diagram according to embodiment 1 of the present application.

[0019] Figure 4 is a hardware fault prediction schematic diagram according to embodiment 1 of the present application.

[0020] Figure 5 is a reinforcement learning and scheduling strategy schematic diagram according to embodiment 1 of the present application. DETAILED DESCRIPTION

[0021] The present disclosure will be further described below in combination with the drawings and embodiments.

[0022] Embodiment 1, Referring to Figure 1 , the present embodiment provides a Huawei Atlas heterogeneous computing resource intelligent scheduling method based on dynamic load perception, which comprises the following steps: S1, as shown, real-time multi-dimensional state perception is performed: real-time state acquisition is performed on hardware resource data, task characteristic data and energy consumption data to provide a basis data source for subsequent steps; Figure 2 ​ The hardware resource data includes: Ascend NPU utilization (including Cube, Vector, and Scalar units), NVIDIA GPU utilization (including Tensor Core, CUDA Core, and SM general-purpose units), memory, bandwidth, and temperature; The task feature data includes: model type (Transformer / CNN / control flow), computational intensity (FLOPs / task), and priority; The energy consumption data includes: power consumption per card / total device (W).

[0023] S2, such as Figure 3 As shown, a hardware feature modeling engine is used to dynamically adapt the task and hardware features (such as the matching between Transformer and Cube Core) through a matching degree matrix. Modeling is performed on various types of basic data to construct the state vectors required for reinforcement learning. The specific steps are as follows: S21. Construct a task-hardware matching matrix for NPU and GPU hardware resource data. The matching degree calculation formula is as follows: (1) In equation (1), This represents the matching degree between a specific task and a hardware unit. The higher the proportion of the task's computational requirements to the hardware capacity, the higher the proportion of hardware resources the task requires and the more suitable it is to be allocated to this hardware unit. This represents the computational load required to execute the task. This indicates the maximum computing power of the hardware unit. The total bandwidth resources required for task execution. Indicates the maximum bandwidth of the hardware unit; and For experience weights.

[0024] Example matching results are shown in Table 1: Table 1 Examples of NPU / GPU AI Unit Matching Degree with Task Type

[0025] S22. Concatenate all matching degrees to obtain a matching degree vector. Concatenate the matching degree vector with other basic data to obtain the state vector required for reinforcement learning. : (2) Wherein, temp represents temperature, mem represents memory, utils represents the utilization rate of NPU and GPU, a total of 6 dimensions (3 kinds of NPU units + 3 kinds of GPU units), power represents energy consumption, and the above is hardware resource data; task_type_embed represents the task-hardware matching degree vector, a total of 6 dimensions (3 kinds of NPU units + 3 kinds of GPU units), task_priority represents task priority, task_flops represents calculation intensity, and task_band represents calculation bandwidth, and the above is task feature data.

[0026] S3, as Figure 4 shown, in order to consider the node burst failure or degradation risk encountered in scheduling tasks, a hardware failure prediction mechanism is introduced, and the results are included in the scheduling strategy to improve system stability and safety, specifically referring to: through the deployment of a lightweight prediction model LightGBM on each computing node, input the temperature sequence of the node in the past 3 minutes, the node memory bandwidth fluctuation rate, the node utilization fluctuation rate, and the node burst restart number, and output the failure risk score; The calculation formula of the failure risk score is: (3) In formula (3), , represents the failure risk score; represents the temperature sequence of the node in the past 3 minutes, represents the node memory bandwidth fluctuation rate, represents the node utilization fluctuation rate, represents the node burst restart number, is an activation function, , , and are weight coefficients; S4, as Figure 5 shown, in order to obtain a scheduling model with global optimization and fault tolerance capability, reinforcement learning and scheduling strategy are adopted: a lightweight deep Q network is used as the model architecture, input the state vector (dimension is 1x19, including 9 dimensions of hardware state + 9 dimensions of task state + 1 dimension of risk score) and the failure risk score, and the final scheduling model is obtained through reinforcement learning training of the reward function in the form of a multi-objective vector, and the task allocation decision is output; The reward function in the form of a multi-objective vector is: (4) Wherein is the delay, is the resource utilization rate, is the node energy consumption, For punishment scheduling hotspot, when the node temperature is higher than the preset threshold 80℃, it is 1, otherwise it is 0.

[0027] The reinforcement learning training process is as follows: a, initialize online Q network and target Q network , set the experience revisit cache size to 10000; b, collect state vector every 5 seconds Input into online Q network, output scheduling action ; the scheduling action refers to task allocation decision, which task is allocated to which computing unit; (5) where is Parameter in greedy strategy, initially 1, gradually reduced with training round number.

[0028] c, if the current node risk score is greater than the preset threshold 0.7, trigger fault tolerance strategy and fault rollback strategy, otherwise execute scheduling action; The fault tolerance strategy refers to backing up the current node to other stable nodes at the same time; the fault rollback strategy refers to re-scheduling the task if there is no feedback within a fixed time (such as 5 seconds).

[0029] d, get reward after executing scheduling action , update network parameters: (6) (7) (8) Where, y is the target Q value, r is the reward, s is the state, a is the action, and Parameters of online Q network and target Q network respectively, B is the training batch size, i is the training step number, is the learning rate.

[0030] Embodiment 2, The embodiment provides a Huawei Atlas heterogeneous computing resource intelligent scheduling system based on dynamic load perception, which comprises: Multi-dimensional state perception module: used for real-time collection of hardware resource data, energy consumption data and task characteristic data, wherein the hardware resource data includes Ascend NPU utilization rate, NVIDIA GPU utilization rate, memory bandwidth and temperature; the energy consumption data includes single card / whole machine power consumption; and the task characteristic data includes model type, calculation intensity and priority; The hardware characteristic modeling engine is configured to construct a task-hardware matching degree matrix based on the task characteristic data and hardware resource data, splice the matching degree matrix with the hardware resource data, energy consumption data, and task characteristic data to obtain a state vector required by reinforcement learning; The fault prediction and fault tolerance awareness scheduling module is configured to input a temperature sequence of a node in the past 3 minutes, a node memory bandwidth fluctuation rate, a node utilization fluctuation rate, and a number of sudden restarts of the node, output a fault risk score, and execute a corresponding fault tolerance strategy according to the fault risk score by using a lightweight prediction model deployed on each computing node; The reinforcement learning and scheduling strategy module is configured to use a lightweight deep Q network as a model architecture, input the state vector and the fault risk score, train by using a reward function in a multi-target vector form, obtain a final scheduling model, and output a task allocation decision.

[0031] Embodiment 3, The embodiment provides a Huawei Atlas heterogeneous computing resource intelligent scheduling method based on dynamic load awareness, which is applied to a private cloud 1024 card Ascend 910B / NVIDIA A100 hybrid cluster, the hardware environment is 512 Ascend 910B cards+512 NVIDIA A100 cards, and the deployment task is to deploy a DeepSeek 70B inference (high priority) and an OCR model training (low priority) in a mixed manner. The specific steps are as follows: 1. Start a multi-dimensional state acquisition module to report hardware temperature and task queue length every 5 seconds; 2. The hardware modeling layer allocates a Transformer task to a Cube / CUDA core and an OCR task to a Vector / Tensor Core according to the matching degree matrix; 3. The DQN scheduler dynamically adjusts the task priority to ensure that the DeepSeek task delay is less than 300 ms; The effects are verified as follows: 1. Resource utilization: The average utilization rate of the Ascend NPU / NVIDIA GPU is more than 82% (the prior art is 60%); 2. Energy consumption cost: The total power consumption of the cluster is reduced by 18%, and more than 20 million yuan of electricity bills are saved per year; 3. Communication throughput: The average throughput rate of the Ascend NPU / NVIDIA GPU is more than 90%; 4. Fault interruption times: reduced from 15 times / hour to 3 times / hour, with an improvement of 80%.

[0032] Obviously, the above embodiments of the present application are only examples for clearly illustrating the technical solutions of the present application, but are not intended to limit the specific implementation of the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A Huawei Atlas heterogeneous computing resource intelligent scheduling method based on dynamic load perception, characterized in that, The method comprises: S1, real-time multi-dimensional state perception is implemented, and hardware resource data, task characteristic data and energy consumption data are collected in real-time state to provide basic data sources for subsequent steps; S2, the task and the hardware characteristics are dynamically adapted through a matching degree matrix, modeling is performed on various basic data, and a state vector required by reinforcement learning is constructed; S3, a hardware fault prediction mechanism is introduced, a light-weight prediction model LightGBM deployed on each computing node is input with temperature sequence, node memory bandwidth fluctuation rate, node utilization fluctuation rate and node burst restart number in a fixed time in the past, and outputs a fault risk score; S4, adopt lightweight deep Q network as model architecture, input the state vector And the failure risk score, through the reward function in the form of multi-objective vector, reinforcement learning training is carried out, and the final scheduling model is obtained, and the task allocation decision is output.

2. The Huawei Atlas heterogeneous computing resource intelligent scheduling method based on dynamic load perception according to claim 1, characterized in that, The hardware resource data comprises NPU utilization rate, GPU utilization rate, memory, bandwidth and temperature; wherein, the NPU comprises Cube, Vector and Scalar units, and the GPU comprises Tensor Core, CUDA Core and SM general unit; The task characteristic data comprises model type, computation intensity and priority; The energy consumption data comprises single card and whole machine power consumption. 3.The Huawei Atlas heterogeneous computing resource intelligent scheduling method based on dynamic load awareness according to claim 1, characterized in that, The specific steps of S2 are as follows: S21, a task-hardware matching degree matrix is constructed for NPU and GPU hardware resource data, and the matching degree calculation formula is as follows: （1） In formula (1), is expressed as the matching degree of a specific task and a hardware unit, and the greater the proportion of the computing requirement of the task to the hardware capability, the higher the proportion of the task to the hardware resource, and the more suitable the task is to be allocated to the hardware unit; is expressed as the computing amount required for task execution, is expressed as the maximum computing capability of the hardware unit, is the total bandwidth resource required for task execution, is expressed as the maximum bandwidth of the hardware unit; and is an empirical weight; S22, splice all matching degrees to obtain a matching degree vector, splice the matching degree vector with other basic data to obtain a state vector required by reinforcement learning : （2） Wherein, temp represents temperature, mem represents memory, utils represents NPU and GPU utilization rate, and power represents energy consumption, which are hardware resource data; task_type_embed represents a task-hardware matching degree vector, which is 6-dimensional; task_priority represents task priority, task_flops represents computation intensity, and task_band represents computation bandwidth, which are task characteristic data. 4.The Huawei Atlas heterogeneous computing resource intelligent scheduling method based on dynamic load awareness according to claim 1, characterized in that, The calculation formula of the fault risk score in S3 is as follows: （3） In formula (3), , represents the fault risk score; represents the temperature sequence of the node in the past 3 minutes, represents the node memory bandwidth fluctuation rate, represents the node utilization fluctuation rate, represents the number of sudden restarts of the node, is an activation function, , , and are weight coefficients.

5. The Huawei Atlas heterogeneous computing resource intelligent scheduling method based on dynamic load perception according to claim 1, characterized in that, The reward function in the form of a multi-objective vector is as follows: （4） wherein is the delay, is the resource utilization, is the node energy consumption, is the punishment dispatching hot spot, which is 1 when the node temperature is higher than the preset threshold 80℃, otherwise 0.

6. The Huawei Atlas heterogeneous computing resource intelligent scheduling method based on dynamic load perception according to claim 5, characterized in that, The reinforcement learning training process is as follows: a. Initialize the online Q-network and the target Q-network Set the experience replay buffer size to 10,000; b. State vector is collected every 5 seconds Input to online Q-network, output schedule action ; The scheduling action refers to a task allocation decision, and the current task is allocated to which computing unit; （5） wherein is a parameter in the -greedy policy, initially 1 and gradually reduced with the number of training rounds; c, when the current node risk score is greater than a preset threshold 0.7, a fault tolerance strategy and a fault rollback strategy are triggered, otherwise the scheduling action is executed; The fault tolerance strategy refers to backing up the current node to other stable nodes at the same time; The fault rollback strategy refers to task rescheduling if there is no feedback within a fixed time of the node; d. obtaining a reward after performing the scheduling action updating the network parameters: （6） （7） （8） where y is the target Q value, r is the reward, s is the state, and a is the action, and are the parameters of the online and target Q networks, respectively, B is the batch size, i is the training step number, is the learning rate.

7. A Huawei Atlas heterogeneous computing resource intelligent scheduling system based on dynamic load perception, the system comprising: A multi-dimensional state perception module: used for collecting hardware resource data, energy consumption data and task characteristic data in real time, wherein the hardware resource data comprises NPU utilization rate, GPU utilization rate, memory bandwidth and temperature; the energy consumption data comprises single card / whole machine power consumption; and the task characteristic data comprises model type, computation intensity and priority; A hardware characteristic modeling engine: used for constructing a task-hardware matching degree matrix based on the task characteristic data and the hardware resource data, splicing the matching degree matrix with the hardware resource data, energy consumption data and task characteristic data to obtain a state vector required by reinforcement learning; and A fault tolerance strategy refers to backing up the current node to other stable nodes at the same time; The fault prediction and fault-tolerant awareness scheduling module is configured to input a temperature sequence of a fixed time in the past, a memory bandwidth fluctuation rate of a node, a node utilization fluctuation rate, and a number of sudden restarts of the node, output a fault risk score, and execute a corresponding fault-tolerant strategy according to the fault risk score by deploying a lightweight prediction model on each computing node. The reinforcement learning and scheduling strategy module is configured to adopt a lightweight deep Q network as a model architecture, input the state vector and the fault risk score, train through a reward function in a form of a multi-objective vector, obtain a final scheduling model, and output a task allocation decision.

Images