Big data mining-based AI server cluster computing power scheduling method

Abstract

The application discloses a computing power scheduling method of an AI server cluster based on big data mining, and comprises the following steps: S1, collecting node load and task demand data; S2, mapping a state and demand vector and calculating a resource hunger index; S3, inputting an improved MOEA / D algorithm, constructing a topological similarity dynamic neighborhood, calculating an affinity score and reducing a weight with adaptive punishment, and outputting a target node; S4, detecting resource conflicts, issuing a transfer instruction for high-priority tasks to occupy, and recording a borrowing and compensation relationship; S5, returning resources and compensation time length after the completion of high-priority tasks, integrating results, and outputting a final scheduling decision; S6, deploying and monitoring, migrating data in case of failure, counting the cost, outputting a closed-loop feedback, and adaptively correcting the weight and the penalty coefficient. The application improves the matching degree, effectively resolves resource conflicts, and realizes efficient and robust scheduling of the cluster.

Description

Technical Field

[0001] This invention relates to the intersection of computer science and artificial intelligence, and in particular to a computing power scheduling method for AI server clusters based on big data mining. Background Technology

[0002] Improved multi-objective evolutionary algorithms, due to their global search and multi-objective equilibrium optimization capabilities under complex constraints, have been widely applied in recent years in fields such as cloud computing resource allocation, heterogeneous cluster scheduling, and intelligent computing orchestration, becoming an important development direction for large-scale AI computing power scheduling. However, in practical applications, AI server cluster scheduling faces many challenges, such as large differences in task requirements, multi-dimensional heterogeneous resources, and strong coupling between communication and computing. The deployment effectiveness of improved multi-objective evolutionary algorithms is still constrained by many factors.

[0003] Currently, most cluster scheduling methods rely on single metrics such as computing power or memory load for matching, making it difficult to fully utilize multi-dimensional state information such as GPU memory usage, inter-node communication bandwidth margin, and task queuing depth. This results in a lack of comprehensive modeling of the real-time carrying capacity of nodes and the true task hunger. Some systems only use static weight vector distance to define the evolutionary neighborhood, ignoring the dynamic impact of communication density and network topology in distributed AI training, which limits the adaptive optimization capability of scheduling strategies in communication bottleneck scenarios. At the same time, the scheduling logic lacks preemption and compensation mechanisms for resource conflicts, making it difficult to provide reliable operational guarantees for long-cycle tasks and affecting the overall availability of cluster resources.

[0004] Furthermore, most existing scheduling methods employ open-loop feedback mechanisms in the runtime state, failing to dynamically adjust the hunger weight and affinity matching penalty coefficient based on scheduling costs such as node failures or migration overhead. This results in some key dimensions performing poorly for a long time without effective correction, making it difficult to adapt to sudden drops in cluster load and risk evolution, severely impacting the practical value and stability of the scheduling system in real-world scenarios.

[0005] Therefore, how to provide a computing power scheduling method for AI server clusters based on big data mining is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a computing power scheduling method for AI server clusters based on big data mining. This invention fully integrates key steps such as multi-dimensional hunger quantification, an improved MOEA / D algorithm, resource conflict relinquishment, and fault migration feedback to construct an intelligent computing power allocation and closed-loop optimization process with communication topology awareness, dynamic priority preemption, and adaptive adjustment of scheduling overhead. Addressing the shortcomings of existing methods that neglect communication coupling and lack preemption compensation mechanisms, this invention introduces an improved MOEA / D algorithm to construct a dynamic neighborhood reconstruction mechanism based on communication topology similarity, replacing the original weighted distance to accurately match communication needs and bandwidth status. Simultaneously, it designs precision relinquishment and duration compensation strategies for high and low priority tasks, and dynamically adjusts weights and penalty coefficients based on scheduling overhead. This invention possesses advantages such as multi-dimensional supply and demand matching, high communication fit through neighborhood evolution, flexible resource conflict relinquishment, and strong adaptive scheduling feedback. It can significantly improve the accuracy and operational stability of computing power allocation in complex cluster environments, thereby effectively solving problems such as weak communication topology awareness, crude resource preemption, and lack of feedback mechanisms in existing scheduling methods.

[0007] The computing power scheduling method for AI server clusters based on big data mining according to embodiments of the present invention includes the following steps: S1. Synchronously collect hardware load data of each node in the AI ​​server cluster and resource requirement data of the AI ​​tasks to be scheduled, and output the original dataset of cluster resources and tasks. S2. Map the cluster resources and hardware load data in the original task dataset to a multi-dimensional resource status vector, map the resource demand data to a task demand vector, calculate the task's thirst index for various computing and storage resources on the node, and output the task resource thirst index set. S3. Input the task resource hunger index set and communication density identifier into the improved MOEA / D algorithm, construct a dynamic neighborhood reconstruction mechanism based on communication topology similarity, calculate affinity matching score, calculate topology similarity to replace the original weight distance in the iteration, determine the trigger of adaptive penalty, calculate the penalty term according to the shortage and iteration state and add it to the aggregation function to reduce weight, search along the reconstructed neighborhood, and output the target node set. S4. For the target node set, detect whether there is a resource conflict between high-priority burst tasks and low-priority long tasks running on the nodes, issue a computing power transfer instruction, control the release of some video memory and computing power for high-priority burst tasks to occupy, record the computing power borrowing relationship and compensation strategy, and output the resource conflict transfer and compensation record. S5. After the high-priority burst task is completed, according to the resource conflict transfer and compensation record, the released resources and additional computing power compensation time are returned to the low-priority long task. The conflict handling results and affinity matching score screening results are integrated to output the final computing power scheduling decision. S6 issues the final computing power scheduling decision, executes the deployment, and monitors the hardware load in real time. When a node failure or computing power drop is detected, a takeover node is selected and the data is transferred. The scheduling overhead is calculated and the scheduling closed-loop feedback data is output. The hunger weight and affinity matching penalty coefficient are adaptively adjusted.

[0008] Optionally, S1 specifically includes: S11. Synchronously read the performance counters and task scheduler queue status of each node through the cluster monitoring agent to obtain GPU computing utilization, video memory usage, inter-node communication bandwidth margin and task queuing depth. S12. Parse the configuration file and submission instructions of the AI ​​task to be scheduled, extract the peak computing power and minimum memory requirements of the task, and compare them with the preset communication-intensive task feature library to assign a communication density identifier to the task. At the same time, calculate the difference between the task deadline and the current submission time and divide it by the estimated total duration of the task to obtain the completion time weight. S13. Normalize the collected GPU computing utilization, video memory usage, inter-node communication bandwidth margin and task queuing depth, and linearly map the values ​​of each hardware load data to between 0 and 1. Normalize the extracted task computing power peak, video memory minimum requirement, communication density identifier and completion time weight, and linearly map the values ​​of each resource requirement data to between 0 and 1. S14. Align and stitch the normalized hardware load data and resource requirement data according to the collection timestamp and task identifier to generate a standardized data form and output the original dataset of cluster resources and tasks.

[0009] Optionally, S2 specifically includes: S21. Arrange the GPU computing utilization, video memory usage, inter-node communication bandwidth margin, and task queuing depth in the cluster resources and task original dataset according to the node identifier, and combine these four values ​​of each node to generate a multi-dimensional resource state vector representing the real-time carrying capacity of each node. S22. Arrange the peak computing power, minimum memory requirement, communication density identifier, and completion time weight of the task in the cluster resources and the original task dataset according to the task identifier, and combine these four values ​​of each task to generate a task requirement vector. S23. For each combination of task and node, subtract the GPU computing utilization rate in the multi-dimensional resource state vector from the peak computing power of the task in the task demand vector to obtain the computing power hunger difference, and subtract the video memory usage from the video memory baseline demand to obtain the video memory hunger difference. S24. Divide the computing power hunger difference by the peak computing power of the task to obtain the computing power hunger ratio, divide the memory hunger difference by the minimum memory requirement to obtain the memory hunger ratio, multiply the communication density indicator by the reciprocal of the inter-node communication bandwidth margin to obtain the communication hunger coefficient, and multiply the completion time weight by the reciprocal of the task queuing depth to obtain the timeliness hunger coefficient. S25. The computing power hunger ratio, video memory hunger ratio, communication hunger coefficient and timeliness hunger coefficient are weighted and summed according to the preset resource weights. The summation result is used as the task's hunger index for node computing and storage resources. All task and node combinations are traversed to output the task resource hunger index set.

[0010] Optionally, S3 specifically includes: S31. The task resource hunger index set and communication density identifier are used as input vectors to import the improved MOEA / D algorithm. The population is initialized according to the number of task and node combinations. Each scheduling individual is assigned a weight vector consisting of computing power weight, memory weight and communication weight. The three objective functions of computing power matching, memory matching and communication matching are decomposed into multiple single-objective scheduling sub-problems. S32. For each scheduling individual, subtract the GPU computing utilization rate in the multi-dimensional resource state vector from the peak computing power in the task demand vector to obtain the remaining computing power difference, subtract the video memory usage from the video memory baseline demand to obtain the remaining video memory difference, multiply the communication density identifier by the communication bandwidth margin between nodes to obtain the communication bandwidth product, and use the remaining computing power difference, remaining video memory difference and communication bandwidth product as computing power fit, video memory fit and communication fit, respectively. S33. Traverse all scheduled individuals and find the maximum values ​​of computing power fit, memory fit, and communication fit, respectively as the ideal points of computing power, memory, and communication. Subtract the ideal point of computing power from the computing power fit of each individual to obtain the computing power difference, subtract the ideal point of memory from the memory fit to obtain the memory difference, and subtract the ideal point of communication from the communication fit to obtain the communication difference. S34. Divide the computing power difference by the computing power weight in the weight vector to obtain the computing power quotient, divide the memory difference by the memory weight to obtain the memory quotient, divide the communication difference by the communication weight to obtain the communication quotient, compare the computing power quotient, memory quotient and communication quotient, extract the quotient with the largest value as the output of the Chebyshev aggregation function, and use the output value as the affinity matching score for task deployment to each candidate node. S35. In the iteration, extract the communication density identifier and the inter-node communication bandwidth margin corresponding to the current scheduling individual and another scheduling individual. Subtract the communication density identifier of the other individual from the communication density identifier of the current individual and take the absolute value as the communication demand distance. Subtract the inter-node communication bandwidth margin of the other individual from the inter-node communication bandwidth margin of the current individual and take the absolute value as the bandwidth state distance. Perform a weighted summation of the communication demand distance and the bandwidth state distance and take the reciprocal to obtain the communication topology similarity. S36. Calculate the communication topology similarity between the current scheduling individual and all other scheduling individuals. Sort all the calculated communication topology similarities in ascending order of value. Select the first preset number of scheduling individuals in the sorting results to form a neighborhood and replace the evolutionary neighborhood calculated based on the weight vector Euclidean distance in the original algorithm. Perform dynamic neighborhood reconstruction based on communication topology similarity. S37. By comparing the reconstructed evolutionary neighborhood, the dynamic penalty term is calculated. Combined with the affinity matching score, the target node set is output.

[0011] Optionally, S37 specifically includes: S371. In the reconstructed evolutionary neighborhood, select neighboring individuals and perform crossover mutation operations with the current individual to generate offspring individuals. That is, generate a random number between 0 and 1 as the crossover probability. When the random number is less than the preset crossover probability, disconnect the node allocation sequence of the current individual and the neighboring individuals at the preset crossover point and exchange the second half of the disconnected sequence to splice it into a new sequence. S372. Then, a random number between 0 and 1 is generated as the mutation probability. When the random number is less than the preset mutation probability, a position is randomly selected in the node allocation sequence of the offspring individual, and the node assigned to the position is replaced with another node randomly selected in the cluster. The inter-node communication bandwidth margin of the candidate node assigned to the offspring individual is read, and the inter-node communication bandwidth margin is compared with the preset bandwidth threshold to determine whether the inter-node communication bandwidth margin is lower than the preset bandwidth threshold. S373. If the inter-node communication bandwidth margin is lower than the preset bandwidth threshold, an adaptive penalty mechanism is triggered. The preset bandwidth threshold is subtracted from the inter-node communication bandwidth margin to obtain the current bandwidth deficit. The current iteration number and the preset total iteration number are read. The current iteration number is divided by the preset total iteration number to obtain the iteration progress ratio. The current bandwidth deficit is multiplied by the iteration progress ratio to calculate the dynamic penalty term. S374. Perform an addition operation on the dynamic penalty term and the output value of the Chebyshev aggregation function, and use the sum as the weighted affinity matching score. Since the smaller the affinity matching score, the higher the node matching priority, the larger the affinity matching score after the dynamic penalty term is added, it means that the node has been penalized and weighted down. Compare the weighted affinity matching score of the offspring with the affinity matching score of the individuals in the evolutionary neighborhood. S375. If the affinity matching score of the offspring individual after weight reduction is less than the affinity matching score of the individual in the evolutionary neighborhood, then replace the individual in the evolutionary neighborhood with the offspring individual, and search for co-evolution along the reconstructed neighborhood until the preset termination condition is reached, and output the target node set.

[0012] Optionally, S4 specifically includes: S41. For the target node set, read the completion time weight and estimated total duration of the AI ​​task to be scheduled. The AI ​​task to be scheduled with a completion time weight greater than the preset time threshold and an estimated total duration less than the preset short task duration threshold is identified as a high-priority burst task. The task with a completion time weight less than or equal to the preset time threshold and an estimated total duration greater than or equal to the preset long task duration threshold is identified as a low-priority long task. S42. Add the peak computing power and minimum memory requirements of high-priority burst tasks on each target node to the current computing power and memory usage of low-priority long tasks running on the target node to obtain the total computing power requirement and the total memory requirement. S43. Subtract the total computing power demand from the total computing power limit of the target node to obtain the computing power overflow. Subtract the total video memory demand from the total video memory limit of the target node to obtain the video memory overflow. If the computing power overflow or video memory overflow is greater than zero, it is determined that there is a resource conflict between high-priority burst tasks and low-priority long tasks. S44. If there is a resource conflict, a computing power transfer instruction is issued to the low-priority long task, reducing the current computing precision of the low-priority long task from the preset initial precision to the preset transfer precision, and compressing the current batch size of the low-priority long task from the preset initial batch to the preset transfer batch. S45. Based on the preset transfer precision after the reduction and the compressed transfer batch, calculate the computing power capacity and video memory capacity released by low-priority long tasks, and allocate the released computing power capacity and video memory capacity for high-priority burst tasks to occupy. S46. Bind the task identifier of high-priority burst tasks with the task identifier of low-priority long tasks to form a computing power borrowing relationship. Perform a multiplication operation between the estimated execution time of high-priority burst tasks and the preset compensation multiplier to obtain an additional computing power compensation time. Combine the computing power borrowing relationship and the computing power compensation time as a compensation strategy. Summarize the computing power borrowing relationship and the compensation strategy to output resource conflict transfer and compensation records.

[0013] Optionally, S5 specifically includes: S51. After the high-priority burst task is completed, read the computing power borrowing relationship in the resource conflict transfer and compensation record, and locate the corresponding low-priority long task according to the computing power borrowing relationship. S52. Read the compensation strategy in the resource conflict transfer and compensation record, extract the computing power compensation duration in the compensation strategy, extend the current scheduling deadline of the low-priority long task by the value of the computing power compensation duration, and return the extended time period as an additional computing power compensation duration to the low-priority long task. S53. Restore the computational precision of the low-priority long task from the preset surrender precision to the preset initial precision, restore the compressed batch size from the preset surrender batch to the preset initial batch, and return the released computing power and video memory capacity to the low-priority long task. S54. Extract all resource conflict judgment results of high-priority burst tasks and low-priority long tasks and integrate them with computing power transfer execution records into conflict handling results. Extract the affinity matching score output by the Chebyshev aggregation function as the affinity matching score filtering result. S55. Align and concatenate the conflict resolution results and affinity matching score filtering results according to the task identifier. Generate a scheduling data table by extending the deadline of low-priority long tasks, restoring the accuracy, and batch overwriting the original task parameters, and use it as the final computing power scheduling decision output.

[0014] Optionally, S6 specifically includes: S61. Issue the final computing power scheduling decision to each node of the AI ​​server cluster to execute task deployment. At the same time, read the GPU computing utilization and video memory usage of each node according to the preset monitoring cycle. Mark the nodes with GPU computing utilization below the preset computing drop threshold or the nodes with video memory usage suddenly dropping to zero as fault nodes. S62. When a faulty node is detected, extract the tasks running on the faulty node as tasks to be migrated. In the AI ​​server cluster, among the remaining nodes other than the faulty node, select nodes whose inter-node communication bandwidth margin is greater than the communication density identifier of the task to be migrated and whose GPU computing utilization and video memory usage are both lower than the preset migration load threshold. Select the node with the smallest affinity matching score from the selected nodes as the takeover node. S63. Allocate computing and storage resources on the takeover node and start the process of the task to be migrated. At the same time, copy and transmit the unfinished computing data and model parameter files of the task to be migrated on the faulty node to the takeover node through the network channel. During the data transmission, the faulty node continues to execute the untransmitted part of the computing task until the data transmission volume of the takeover node and the remaining computing volume of the faulty node are completed synchronously. S64. Calculate the additional computing power loss time and network transmission time caused by node failure and task migration. Perform an addition operation on the additional computing power loss time and network transmission time to obtain the scheduling overhead. Combine the scheduling overhead, the node identifier of the faulty node and the node identifier of the takeover node to generate scheduling closed-loop feedback data. S65. Read the scheduling overhead in the scheduling closed-loop feedback data. When the scheduling overhead is greater than the preset overhead warning threshold, add the preset weight correction step size to the values ​​of computing power weight and memory weight in the preset resource weights. Subtract twice the preset weight correction step size from the value of communication weight to adaptively correct the hunger weight. At the same time, add the preset threshold correction step size to the value of the preset bandwidth threshold to adaptively correct the affinity matching penalty coefficient.

[0015] The beneficial effects of this invention are: This invention addresses the issues of coarse task supply-demand matching and weak communication topology awareness in AI clusters by constructing a multi-dimensional resource state vector and task demand vector. It employs a hunger index calculation for multi-dimensional quantification of supply and demand, and decomposes the single-objective scheduling sub-problem by combining computing power fit, memory fit, and communication fit, outputting a task resource hunger index set. By inputting this task resource hunger index set into an improved MOEA / D algorithm, it addresses the problem of traditional neighborhood search ignoring dynamic communication features by constructing a dynamic neighborhood reconstruction mechanism based on communication topology similarity, combining Chebyshev aggregation function and bandwidth deficit self-regulation. The algorithm calculates affinity matching scores based on adaptive penalties and outputs a target node set. During resource conflict handling, to address the issue of high-priority tasks preempting resources and causing interruptions to low-priority long-running tasks, it combines reduced execution precision of computing power transfer instructions with batch compression to release resources for high-priority tasks. It also records computing power borrowing relationships and compensation strategies, returning resources and compensation durations after each task completes. Furthermore, by real-time monitoring of hardware load and statistical scheduling overhead, it dynamically adjusts the hunger weight and affinity matching penalty coefficient, selecting takeover nodes for faulty nodes and transferring data to complete fault-tolerant migration. Ultimately, it achieves precise supply-demand matching, dynamic communication alignment, elastic resource transfer, and self-evolving scheduling within the AI ​​server cluster, effectively improving the affinity matching accuracy of computing power scheduling, the smoothness of task preemption, and the adaptive robustness of the cluster system. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the computing power scheduling method for AI server clusters based on big data mining proposed in this invention; Figure 2 This is a flowchart of the affinity matching scheduling process proposed in this invention, based on multidimensional hunger quantification and improved MOEA / D algorithm. Figure 3 This is a flowchart of the resource conflict transfer compensation and fault migration closed-loop feedback process proposed in this invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figures 1-3 The computing power scheduling method for AI server clusters based on big data mining includes the following steps: S1. Synchronously collect hardware load data and resource requirement data of AI tasks to be scheduled from each node in the AI ​​server cluster. The hardware load data includes GPU computing utilization, video memory usage, inter-node communication bandwidth margin and task queuing depth. The resource requirement data includes task computing power peak, video memory minimum requirement, communication density indicator and completion time weight. Output the cluster resources and task raw dataset. S2. Map the hardware load data in the cluster resources and the original task dataset to a multi-dimensional resource state vector representing the real-time carrying capacity of each node, map the resource demand data to a task demand vector representing the task running characteristics, and calculate the task's thirst index for various computing and storage resources of the node based on the multi-dimensional resource state vector and the task demand vector, and output the task resource thirst index set. S3. Input the task resource hunger index set and communication density identifier into the improved MOEA / D algorithm to construct a dynamic neighborhood reconstruction mechanism based on communication topology similarity. Decompose computing power, memory and communication fit into single-objective scheduling sub-problems. Calculate affinity matching score based on Chebyshev aggregation function. In the iteration, calculate topology similarity based on communication density identifier and bandwidth margin to replace the original weight distance, dynamically reconstruct the evolutionary neighborhood. When the bandwidth margin of a candidate node is lower than the threshold, trigger an adaptive penalty. Calculate the penalty term according to the deficit and iteration state and add it to the aggregation function to reduce weight. Search along the reconstructed neighborhood to output the target node set. S4. For the target node set, detect whether there is a resource conflict between high-priority burst tasks and low-priority long tasks running on the nodes. If there is a conflict, issue a computing power transfer instruction to the low-priority long task, control the release of some video memory and computing power for high-priority burst tasks to be occupied by reducing precision or compressing batch size, record the computing power borrowing relationship and compensation strategy, and output the resource conflict transfer and compensation record. S5. After the high-priority burst task is completed, according to the resource conflict transfer and compensation record, the released resources and additional computing power compensation time are returned to the low-priority long task. The conflict handling results and affinity matching score screening results are integrated to output the final computing power scheduling decision. S6 issues the final computing power scheduling decision, executes the deployment, and monitors the hardware load in real time. When a node failure or computing power drop is detected, a takeover node is selected and the data is transferred. During the migration, the original node continues to compute. The scheduling overhead is statistically analyzed and the scheduling closed-loop feedback data is output. The hunger weight and affinity matching penalty coefficient are adaptively adjusted.

[0019] This invention significantly improves the accuracy and robustness of AI cluster computing power scheduling. By mapping multi-dimensional resource states to task demand vectors, it achieves unified quantitative modeling of node capacity and task characteristics, enhancing the ability to express and match complex supply and demand relationships. During scheduling decisions, a dynamic neighborhood reconstruction mechanism based on communication topology similarity, constructed using an improved MOEA / D algorithm, effectively avoids network congestion in traditional methods, ensuring tasks are accurately matched to high-affinity nodes. When faced with high-priority, sudden task preemption, a flexible transfer and compensation mechanism is used to peacefully reallocate resources by reducing precision or compressing batches. Resources are returned and compensation time is provided after the high-priority task is completed, fundamentally resolving resource conflicts and ensuring the continuity of long-cycle tasks. In fault scenarios, a seamless switching strategy involving the original node continuing computation and the takeover node synchronously migrating greatly reduces the risk of task interruption. Simultaneously, a closed-loop feedback mechanism based on scheduling overhead adaptively adjusts the hunger weight and penalty coefficient, enabling the scheduling strategy to continuously evolve. This invention effectively breaks through the technical bottlenecks of static allocation and brute-force preemption, demonstrating extremely high resource utilization and fault tolerance in heterogeneous computing scenarios, and significantly improving the intelligent level of cluster dynamic scheduling, conflict resolution and fault self-healing.

[0020] In this embodiment, S1 specifically includes: S11. Synchronously read the performance counters and task scheduler queue status of each node through the cluster monitoring agent to obtain GPU computing utilization, video memory usage, inter-node communication bandwidth margin and task queuing depth. S12. Parse the configuration file and submission instructions of the AI ​​task to be scheduled to extract the peak computing power and minimum memory requirements of the task, and assign a communication density identifier to the task by comparing it with the preset communication-intensive task feature library. At the same time, calculate the difference between the task deadline and the current submission time and divide it by the estimated total duration of the task to obtain the completion time weight. The preset communication-intensive task feature library is a set of regular expression matching rules containing keywords of distributed training and gradient synchronization. S13. Normalize the collected GPU computing utilization, video memory usage, inter-node communication bandwidth margin and task queuing depth, and linearly map the values ​​of each hardware load data to between 0 and 1. Normalize the extracted task computing power peak, video memory minimum requirement, communication density identifier and completion time weight, and linearly map the values ​​of each resource requirement data to between 0 and 1. S14. Align and stitch the normalized hardware load data and resource requirement data according to the collection timestamp and task identifier to generate a standardized data form and output the original dataset of cluster resources and tasks.

[0021] In this embodiment, S2 specifically includes: S21. Arrange the GPU computing utilization, video memory usage, inter-node communication bandwidth margin, and task queuing depth in the cluster resources and task original dataset according to the node identifier, and combine these four values ​​of each node to generate a multi-dimensional resource state vector representing the real-time carrying capacity of each node. S22. Arrange the peak computing power, minimum memory requirement, communication density identifier, and completion time weight of the task in the cluster resources and the original task dataset according to the task identifier, and combine these four values ​​of each task to generate a task requirement vector that represents the task operation characteristics. S23. For each combination of task and node, subtract the GPU computing utilization rate in the multi-dimensional resource state vector from the peak computing power of the task in the task demand vector to obtain the computing power hunger difference, and subtract the video memory usage from the video memory baseline demand to obtain the video memory hunger difference. S24. Divide the computing power hunger difference by the peak computing power of the task to obtain the computing power hunger ratio, divide the memory hunger difference by the minimum memory requirement to obtain the memory hunger ratio, multiply the communication density indicator by the reciprocal of the inter-node communication bandwidth margin to obtain the communication hunger coefficient, and multiply the completion time weight by the reciprocal of the task queuing depth to obtain the timeliness hunger coefficient. S25. The computing power hunger ratio, video memory hunger ratio, communication hunger coefficient, and timeliness hunger coefficient are weighted and summed according to preset resource weights. The summation result is used as the task's hunger index for node computing and storage resources. All task and node combinations are traversed to output the task resource hunger index set. In the preset resource weights, the computing power weight is 0.4, the video memory weight is 0.3, the communication weight is 0.2, and the timeliness weight is 0.1.

[0022] In this embodiment, S3 specifically includes: S31. The task resource hunger index set and communication density identifier are used as input vectors to import the improved MOEA / D algorithm. The population is initialized according to the number of task and node combinations. Each scheduling individual is assigned a weight vector consisting of computing power weight, memory weight and communication weight. The three objective functions of computing power matching, memory matching and communication matching are decomposed into multiple single-objective scheduling sub-problems. S32. For each scheduling individual, subtract the GPU computing utilization rate in the multi-dimensional resource state vector from the peak computing power in the task demand vector to obtain the remaining computing power difference, subtract the video memory usage from the video memory baseline demand to obtain the remaining video memory difference, multiply the communication density identifier by the communication bandwidth margin between nodes to obtain the communication bandwidth product, and use the remaining computing power difference, remaining video memory difference and communication bandwidth product as computing power fit, video memory fit and communication fit, respectively. S33. Traverse all scheduled individuals and find the maximum values ​​of computing power fit, memory fit, and communication fit, respectively as the ideal points of computing power, memory, and communication. Subtract the ideal point of computing power from the computing power fit of each individual to obtain the computing power difference, subtract the ideal point of memory from the memory fit to obtain the memory difference, and subtract the ideal point of communication from the communication fit to obtain the communication difference. S34. Divide the computing power difference by the computing power weight in the weight vector to obtain the computing power quotient, divide the memory difference by the memory weight to obtain the memory quotient, divide the communication difference by the communication weight to obtain the communication quotient, compare the computing power quotient, memory quotient and communication quotient, extract the quotient with the largest value as the output of the Chebyshev aggregation function, and use the output value as the affinity matching score for task deployment to each candidate node. S35. In the iteration, extract the communication density identifier and the inter-node communication bandwidth margin corresponding to the current scheduling individual and another scheduling individual. Subtract the communication density identifier of the other individual from the communication density identifier of the current individual and take the absolute value as the communication demand distance. Subtract the inter-node communication bandwidth margin of the other individual from the inter-node communication bandwidth margin of the current individual and take the absolute value as the bandwidth state distance. Perform a weighted summation of the communication demand distance and the bandwidth state distance and take the reciprocal to obtain the communication topology similarity. S36. Calculate the communication topology similarity between the current scheduling individual and all other scheduling individuals. Sort all the calculated communication topology similarities in ascending order of value. Select the scheduling individuals with the first preset number of sorting results to form a neighborhood and replace the evolutionary neighborhood calculated based on the weight vector Euclidean distance in the original algorithm. Perform dynamic neighborhood reconstruction based on communication topology similarity. The preset number is 10. S37. By comparing the reconstructed evolutionary neighborhood, the dynamic penalty term is calculated. Combined with the affinity matching score, the target node set is output.

[0023] In this embodiment, S37 specifically includes: S371. In the reconstructed evolutionary neighborhood, select neighboring individuals and perform crossover mutation operations with the current individual to generate offspring individuals. That is, generate a random number between 0 and 1 as the crossover probability. When the random number is less than the preset crossover probability, disconnect the node allocation sequence of the current individual and the neighboring individuals at the preset crossover point and exchange the second half of the disconnected sequence to splice it into a new sequence. The preset crossover probability is 0.9. S372. Subsequently, a random number between 0 and 1 is generated as the mutation probability. When the random number is less than the preset mutation probability, a position is randomly selected in the node allocation sequence of the offspring individual, and the node assigned to the position is replaced with another node randomly selected in the cluster. The inter-node communication bandwidth margin of the candidate node assigned to the offspring individual is read, and the inter-node communication bandwidth margin is compared with the preset bandwidth threshold to determine whether the inter-node communication bandwidth margin is lower than the preset bandwidth threshold. The preset mutation probability is 0.1, and the preset bandwidth threshold is 0.15. S373. If the inter-node communication bandwidth margin is lower than the preset bandwidth threshold, an adaptive penalty mechanism is triggered. The current bandwidth deficit is obtained by subtracting the inter-node communication bandwidth margin from the preset bandwidth threshold. The current iteration number and the preset total iteration number are read. The current iteration number is divided by the preset total iteration number to obtain the iteration progress ratio. The current bandwidth deficit is multiplied by the iteration progress ratio to calculate the dynamic penalty term. The preset total iteration number is 200. S374. Perform an addition operation on the dynamic penalty term and the output value of the Chebyshev aggregation function, and use the sum as the weighted affinity matching score. Since the smaller the affinity matching score, the higher the node matching priority, the larger the affinity matching score after the dynamic penalty term is added, it means that the node has been penalized and weighted down. Compare the weighted affinity matching score of the offspring with the affinity matching score of the individuals in the evolutionary neighborhood. S375. If the affinity matching score of the offspring individual after weight reduction is less than the affinity matching score of the individual in the evolutionary neighborhood, then the offspring individual replaces the individual in the evolutionary neighborhood, and the co-evolutionary search along the reconstructed neighborhood continues until a preset termination condition is reached, and the target node set is output. The preset termination condition is that the number of iterations reaches the preset total number of iterations or the population has not produced a better affinity matching score for 30 consecutive generations.

[0024] This invention introduces an improved MOEA / D algorithm combined with dynamic neighborhood reconstruction and an adaptive penalty mechanism to achieve multi-dimensional objective optimization and communication congestion avoidance in AI cluster computing power scheduling. It decomposes computing power, memory, and communication compatibility into single-objective sub-problems, and accurately quantifies the affinity matching priority of tasks to nodes through difference quotient calculation and Chebyshev aggregation functions. During iteration, communication topology similarity is calculated based on communication needs and bandwidth status, replacing the traditional weighted Euclidean distance dynamic reconstruction of the evolutionary neighborhood. This prioritizes information exchange between individuals with similar communication needs, significantly accelerating local convergence. Adaptive penalties are triggered for candidate nodes with bandwidth reserves below a threshold, dynamically calculating penalty terms based on bandwidth deficit and iteration progress, and adding them to the aggregation function for weight reduction. This invention can effectively avoid network congestion nodes under conditions of heterogeneous resource competition and high communication load, ensuring the dual matching accuracy of computing power and communication, and significantly improving the optimization efficiency and global resource adaptability of scheduling decisions.

[0025] The improved MOEA / D algorithm of this invention is similar to the original MOEA / D algorithm in that both retain the core architecture of multi-objective evolution based on decomposition. That is, the multi-objective optimization problem is decomposed into multiple single-objective sub-problems through weight vectors, and the Chebyshev aggregation function is used to evaluate the performance of individuals on the sub-problems. At the same time, the cooperative evolution and information interaction between individuals are carried out based on the neighborhood structure, and both generate offspring through crossover and mutation operations to expand the search space.

[0026] The difference lies in that this invention breaks the limitation of the original MOEA / D algorithm, which statically constructs the evolutionary neighborhood based on Euclidean distance of weight vectors, and introduces a communication awareness and dynamic penalty mechanism. While the original algorithm directly calculates the neighborhood using weighted distances, this invention, in steps S35 and S36, extracts communication density identifiers and bandwidth margins to calculate communication demand distance and bandwidth state distance, then calculates the communication topology similarity by weighted summation and taking the reciprocal, and uses this to dynamically reconstruct the evolutionary neighborhood instead of Euclidean distance. Furthermore, in steps S371 to S374, the original algorithm lacks dynamic intervention for inferior nodes, while this invention introduces a bandwidth threshold determination after the mutation operation. When the bandwidth margin is lower than the threshold, a dynamic penalty term is calculated based on the current bandwidth deficit and the iteration progress ratio, and then superimposed on the output value of the Chebyshev aggregation function for penalized weight reduction.

[0027] Based on the above improvements, the beneficial effects of this invention are that by reconstructing the neighborhood through communication topology similarity, individuals with similar communication needs and bandwidth status are given priority in interaction, breaking the limitation of the original static neighborhood ignoring the communication topology, and significantly accelerating the convergence of solutions with high communication matching degree; the dynamic penalty mechanism can adaptively increase the penalty intensity on congested nodes in the later stage of iteration, accurately avoid nodes with insufficient bandwidth, effectively curb communication congestion, and improve the global adaptability and robustness of scheduling decisions in heterogeneous clusters.

[0028] In this embodiment, S4 specifically includes: S41. For the target node set, read the completion time weight and estimated total duration of the AI ​​tasks to be scheduled. Determine the AI ​​tasks to be scheduled that have a completion time weight greater than a preset time threshold and an estimated total duration less than a preset short task duration threshold as high-priority burst tasks. Determine the tasks in operation that have a completion time weight less than or equal to a preset time threshold and an estimated total duration greater than or equal to a preset long task duration threshold as low-priority long tasks. The preset time threshold is 0.8, the preset short task duration threshold is 2 hours, and the preset long task duration threshold is 24 hours. S42. Add the peak computing power and minimum memory requirements of high-priority burst tasks on each target node to the current computing power and memory usage of low-priority long tasks running on the target node to obtain the total computing power requirement and the total memory requirement. S43. Subtract the total computing power demand from the total computing power limit of the target node to obtain the computing power overflow. Subtract the total video memory demand from the total video memory limit of the target node to obtain the video memory overflow. If the computing power overflow or video memory overflow is greater than zero, it is determined that there is a resource conflict between high-priority burst tasks and low-priority long tasks. The total computing power limit is the preset peak computing power of the node hardware, and the total video memory limit is the preset total video memory of the node hardware. S44. If a resource conflict exists, a computing power transfer instruction is issued to the low-priority long task, reducing the current computing precision of the low-priority long task from the preset initial precision to the preset transfer precision, and compressing the current batch size of the low-priority long task from the preset initial batch to the preset transfer batch. The preset initial precision is FP32, the preset transfer precision is FP16, the preset initial batch is 128, and the preset transfer batch is 32. S45. Based on the preset transfer precision after the reduction and the compressed transfer batch, calculate the computing power capacity and video memory capacity released by low-priority long tasks, and allocate the released computing power capacity and video memory capacity for high-priority burst tasks to occupy. S46. The task identifiers of high-priority burst tasks and low-priority long tasks are bound together as a computing power borrowing relationship. The estimated execution time of high-priority burst tasks is multiplied by a preset compensation multiplier to obtain an additional computing power compensation time. The computing power borrowing relationship and the computing power compensation time are combined as a compensation strategy. The computing power borrowing relationship and the compensation strategy are summarized to output resource conflict transfer and compensation records. The preset compensation multiplier is 1.5.

[0029] In this embodiment, S5 specifically includes: S51. After the high-priority burst task is completed, read the computing power borrowing relationship in the resource conflict transfer and compensation record, and locate the corresponding low-priority long task according to the computing power borrowing relationship. S52. Read the compensation strategy in the resource conflict transfer and compensation record, extract the computing power compensation duration in the compensation strategy, extend the current scheduling deadline of the low-priority long task by the value of the computing power compensation duration, and return the extended time period as an additional computing power compensation duration to the low-priority long task. S53. Restore the computational precision of the low-priority long task from the preset surrender precision to the preset initial precision, restore the compressed batch size from the preset surrender batch to the preset initial batch, and return the released computing power and video memory capacity to the low-priority long task. S54. Extract all resource conflict judgment results of high-priority burst tasks and low-priority long tasks and integrate them with computing power transfer execution records into conflict handling results. Extract the affinity matching score output by the Chebyshev aggregation function as the affinity matching score filtering result. S55. Align and concatenate the conflict resolution results and affinity matching score filtering results according to the task identifier. Generate a scheduling data table containing task allocation nodes, precision configuration, batch configuration and deadline time by extending the deadline time of low priority long tasks and restoring the original task parameters. This table will be used as the final computing power scheduling decision output.

[0030] In this embodiment, S6 specifically includes: S61. Issue the final computing power scheduling decision to each node of the AI ​​server cluster to execute task deployment. At the same time, read the GPU computing utilization and video memory usage of each node according to the preset monitoring cycle. Mark the nodes with GPU computing utilization below the preset computing drop threshold or the nodes with video memory usage suddenly dropping to zero as fault nodes. The preset monitoring cycle is 5 seconds and the preset computing drop threshold is 0.2. S62. When a faulty node is detected, the tasks running on the faulty node are extracted as tasks to be migrated. Among the remaining nodes in the AI ​​server cluster excluding the faulty node, nodes with inter-node communication bandwidth margin greater than the communication density identifier of the task to be migrated and GPU computing utilization and video memory usage both lower than a preset migration load threshold are selected. From the selected nodes, the node with the smallest affinity matching score is selected as the takeover node. The preset migration load threshold is 0.5. S63. Allocate computing and storage resources on the takeover node and start the process of the task to be migrated. At the same time, copy and transmit the unfinished computing data and model parameter files of the task to be migrated on the faulty node to the takeover node through the network channel. During the data transmission, the faulty node continues to execute the untransmitted part of the computing task until the data transmission volume of the takeover node and the remaining computing volume of the faulty node are completed synchronously. S64. Calculate the additional computing power loss time and network transmission time caused by node failure and task migration. Perform an addition operation on the additional computing power loss time and network transmission time to obtain the scheduling overhead. Combine the scheduling overhead, the node identifier of the faulty node and the node identifier of the takeover node to generate scheduling closed-loop feedback data. S65. Read the scheduling overhead in the scheduling closed-loop feedback data. When the scheduling overhead is greater than the preset overhead warning threshold, add the preset weight correction step size to the values ​​of computing power weight and memory weight in the preset resource weights. Subtract twice the preset weight correction step size from the value of communication weight to adaptively correct the hunger weight. At the same time, add the preset threshold correction step size to the value of the preset bandwidth threshold to adaptively correct the affinity matching penalty coefficient. The preset overhead warning threshold is 0.2 times the estimated total duration of the task. The preset weight correction step size is 0.05. The preset threshold correction step size is 0.05.

[0031] Example 1: To verify the feasibility of this invention in the field of intelligent scheduling and dynamic resource management of AI server cluster computing power, this invention was deployed in the intelligent computing power operation platform of a national supercomputing center. This platform carries core services such as large-scale weather forecasting model training, autonomous driving algorithm simulation, and distributed inference of medical images, connecting to eight heterogeneous computing clusters and processing an average of 6.5PB of data per day. In this complex and heterogeneous computing environment, typical scheduling challenges include: weather and simulation tasks have extremely high communication density, easily leading to network congestion between nodes; inference tasks have sudden occurrences, with instantaneous peaks often causing conflicts between computing power and memory resources; furthermore, training interruptions and model crashes caused by single-node hardware failures are also frequent. Traditional cluster scheduling systems mainly rely on static resource quotas and round-robin allocation strategies, failing to perceive the matching degree between task communication topology and node bandwidth. In resource conflicts, they often adopt a crude forced termination mechanism and lack smooth migration capabilities under fault conditions, resulting in low overall cluster resource utilization, severe task queuing backlog, and delayed response to high-priority tasks.

[0032] In practical deployment, the method of this invention first uses monitoring agents deployed on each node to collect multi-dimensional data in real time, such as GPU computing utilization, memory usage, inter-node communication bandwidth reserves, and task queuing depth. This data is then combined with the peak computing power, minimum memory usage, communication density indicators, and completion time weights of the tasks to be scheduled. Through normalization and weighted summation, a multi-dimensional hunger index set is generated to accurately quantify the complex supply and demand state of the cluster. Subsequently, the hunger index is input into the improved MOEA / D algorithm, breaking away from the traditional static neighborhood search logic based on weighted distance. A dynamic neighborhood reconstruction mechanism based on communication topology similarity is constructed, allowing individuals with similar topologies and complementary communication needs to exchange information preferentially during evolution. Simultaneously, by combining Chebyshev aggregation functions and adaptive bandwidth deficit penalty terms, a target node set that balances computing power, memory usage, and communication compatibility is output, fundamentally alleviating the problem of computing power idleness caused by communication congestion. To address resource conflicts caused by sudden tasks, this invention abandons the traditional task circuit breaker mechanism and introduces a resource elastic transfer strategy. When a computing power transfer instruction is issued, the computational precision of low-priority long tasks is automatically reduced to FP16, and the batch size is compressed to 32, freeing up resources for high-priority tasks. The computing power borrowing relationship and compensation duration are recorded. After the high-priority task is completed, the deadline of the low-priority task is automatically extended, and its precision and batch size are restored, achieving peaceful resource transfer. Regarding fault tolerance, when a drop in node computing power or a sudden decrease in memory is detected, a low-load, high-bandwidth node is immediately selected as the takeover node. Through incomplete data synchronization and parameter replication, seamless task migration is achieved. Simultaneously, migration scheduling overhead is statistically analyzed, and the hunger weight and penalty coefficient are dynamically adjusted, forming a closed-loop self-evolving scheduling system. Table 1 below shows the core performance comparison data between the method of this invention and the original scheduling system in various typical computing power task scheduling during a three-month trial period: Table 1. Comparison of core performance data between the present invention and traditional methods.

[0033] Based on the comparative data shown in Table 1, it can be seen that the computing power scheduling method based on the improved MOEA / D algorithm and closed-loop feedback proposed in this invention has significant performance advantages over the traditional static scheduling method in terms of dynamic management and control of resources in heterogeneous clusters. In particular, it has achieved comprehensive improvement in key indicators such as queuing response speed, communication congestion avoidance, resource conflict resolution, and fault tolerance and recovery.

[0034] In terms of scheduling efficiency and communication optimization, this invention significantly shortens task queuing time and curbs network congestion through multi-dimensional hunger quantification and dynamic neighborhood reconstruction mechanisms. In four typical scenarios, the average queuing time has plummeted from hundreds of seconds in traditional methods to less than 100 seconds. For example, in the "medical image inference" scenario, the time has been reduced from 112.7 seconds to 9.8 seconds. Meanwhile, traditional methods suffer from congestion rates as high as 22.5% and 28.6% in "meteorological model training" and "large-scale language model" due to the lack of communication topology awareness. In contrast, this invention reduces the congestion rate to 3.1% and 5.3% respectively through affinity matching scheduling, effectively releasing the constraint of network bandwidth on computing power.

[0035] In terms of resource conflict handling and fault migration, this invention demonstrates exceptional flexibility and robustness. When faced with high-priority task preemption, traditional methods have a 0% success rate in yielding, while this invention, through precision reduction and compensation mechanisms, generally achieves a yield success rate exceeding 91%. Simultaneously, while traditional methods have a fault migration success rate of less than 55%, this invention, through rapid sensing and optimal takeover node selection, raises the migration success rate to over 90%, ensuring the continuity of long-cycle tasks.

[0036] In terms of macro output, the on-time completion rate of tasks in all scenarios is consistently above 93%, far exceeding the 60%-80% range of traditional methods; the average resource utilization rate of nodes has also jumped from the inefficient range of 55%-65% to the efficient range of over 88%, truly realizing the ultimate extraction and efficient flow of computing power.

[0037] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A computing power scheduling method for AI server clusters based on big data mining, characterized in that, The steps include the following: S1. Synchronously collect hardware load data of each node in the AI ​​server cluster and resource requirement data of the AI ​​tasks to be scheduled, and output the original dataset of cluster resources and tasks. S2. Map the cluster resources and hardware load data in the original task dataset to a multi-dimensional resource status vector, map the resource demand data to a task demand vector, calculate the task's thirst index for various computing and storage resources on the node, and output the task resource thirst index set. S3. Input the task resource hunger index set and communication density identifier into the improved MOEA / D algorithm, construct a dynamic neighborhood reconstruction mechanism based on communication topology similarity, calculate affinity matching score, calculate topology similarity to replace the original weight distance in the iteration, determine the trigger of adaptive penalty, calculate the penalty term according to the shortage and iteration state and add it to the aggregation function to reduce weight, search along the reconstructed neighborhood, and output the target node set. S4. For the target node set, detect whether there is a resource conflict between high-priority burst tasks and low-priority long tasks running on the nodes, issue a computing power transfer instruction, control the release of some video memory and computing power for high-priority burst tasks to occupy, record the computing power borrowing relationship and compensation strategy, and output the resource conflict transfer and compensation record. S5. After the high-priority burst task is completed, according to the resource conflict transfer and compensation record, the released resources and additional computing power compensation time are returned to the low-priority long task. The conflict handling results and affinity matching score screening results are integrated to output the final computing power scheduling decision. S6 issues the final computing power scheduling decision, executes the deployment, and monitors the hardware load in real time. When a node failure or computing power drop is detected, a takeover node is selected and the data is transferred. The scheduling overhead is calculated and the scheduling closed-loop feedback data is output. The hunger weight and affinity matching penalty coefficient are adaptively adjusted.

2. The computing power scheduling method for AI server clusters based on big data mining according to claim 1, characterized in that, S1 specifically includes: S11. Synchronously read the performance counters and task scheduler queue status of each node through the cluster monitoring agent to obtain GPU computing utilization, video memory usage, inter-node communication bandwidth margin and task queuing depth. S12. Parse the configuration file and submission instructions of the AI ​​task to be scheduled, extract the peak computing power and minimum memory requirements of the task, and compare them with the preset communication-intensive task feature library to assign a communication density identifier to the task. At the same time, calculate the difference between the task deadline and the current submission time and divide it by the estimated total duration of the task to obtain the completion time weight. S13. Normalize the collected GPU computing utilization, video memory usage, inter-node communication bandwidth margin and task queuing depth, and linearly map the values ​​of each hardware load data to between 0 and 1. Normalize the extracted task computing power peak, video memory minimum requirement, communication density identifier and completion time weight, and linearly map the values ​​of each resource requirement data to between 0 and 1. S14. Align and stitch the normalized hardware load data and resource requirement data according to the collection timestamp and task identifier to generate a standardized data form and output the original dataset of cluster resources and tasks.

3. The computing power scheduling method for AI server clusters based on big data mining according to claim 1, characterized in that, S2 specifically includes: S21. Arrange the GPU computing utilization, video memory usage, inter-node communication bandwidth margin, and task queuing depth in the cluster resources and task original dataset according to the node identifier, and combine these four values ​​of each node to generate a multi-dimensional resource state vector representing the real-time carrying capacity of each node. S22. Arrange the peak computing power, minimum memory requirement, communication density identifier, and completion time weight of the task in the cluster resources and the original task dataset according to the task identifier, and combine these four values ​​of each task to generate a task requirement vector. S23. For each combination of task and node, subtract the GPU computing utilization rate in the multi-dimensional resource state vector from the peak computing power of the task in the task demand vector to obtain the computing power hunger difference, and subtract the video memory usage from the video memory baseline demand to obtain the video memory hunger difference. S24. Divide the computing power hunger difference by the peak computing power of the task to obtain the computing power hunger ratio, divide the memory hunger difference by the minimum memory requirement to obtain the memory hunger ratio, multiply the communication density indicator by the reciprocal of the inter-node communication bandwidth margin to obtain the communication hunger coefficient, and multiply the completion time weight by the reciprocal of the task queuing depth to obtain the timeliness hunger coefficient. S25. The computing power hunger ratio, video memory hunger ratio, communication hunger coefficient and timeliness hunger coefficient are weighted and summed according to the preset resource weights. The summation result is used as the task's hunger index for node computing and storage resources. All task and node combinations are traversed to output the task resource hunger index set.

4. The computing power scheduling method for AI server clusters based on big data mining according to claim 1, characterized in that, S3 specifically includes: S31. The task resource hunger index set and communication density identifier are used as input vectors to import the improved MOEA / D algorithm. The population is initialized according to the number of task and node combinations. Each scheduling individual is assigned a weight vector consisting of computing power weight, memory weight and communication weight. The three objective functions of computing power matching, memory matching and communication matching are decomposed into multiple single-objective scheduling sub-problems. S32. For each scheduling individual, subtract the GPU computing utilization rate in the multi-dimensional resource state vector from the peak computing power in the task demand vector to obtain the remaining computing power difference, subtract the video memory usage from the video memory baseline demand to obtain the remaining video memory difference, multiply the communication density identifier by the communication bandwidth margin between nodes to obtain the communication bandwidth product, and use the remaining computing power difference, remaining video memory difference and communication bandwidth product as computing power fit, video memory fit and communication fit, respectively. S33. Traverse all scheduled individuals and find the maximum values ​​of computing power fit, memory fit, and communication fit, respectively as the ideal points of computing power, memory, and communication. Subtract the ideal point of computing power from the computing power fit of each individual to obtain the computing power difference, subtract the ideal point of memory from the memory fit to obtain the memory difference, and subtract the ideal point of communication from the communication fit to obtain the communication difference. S34. Divide the computing power difference by the computing power weight in the weight vector to obtain the computing power quotient, divide the memory difference by the memory weight to obtain the memory quotient, divide the communication difference by the communication weight to obtain the communication quotient, compare the computing power quotient, memory quotient and communication quotient, extract the quotient with the largest value as the output of the Chebyshev aggregation function, and use the output value as the affinity matching score for task deployment to each candidate node. S35. In the iteration, extract the communication density identifier and the inter-node communication bandwidth margin corresponding to the current scheduling individual and another scheduling individual. Subtract the communication density identifier of the other individual from the communication density identifier of the current individual and take the absolute value as the communication demand distance. Subtract the inter-node communication bandwidth margin of the other individual from the inter-node communication bandwidth margin of the current individual and take the absolute value as the bandwidth state distance. Perform a weighted summation of the communication demand distance and the bandwidth state distance and take the reciprocal to obtain the communication topology similarity. S36. Calculate the communication topology similarity between the current scheduling individual and all other scheduling individuals. Sort all the calculated communication topology similarities in ascending order of value. Select the first preset number of scheduling individuals in the sorting results to form a neighborhood and replace the evolutionary neighborhood calculated based on the weight vector Euclidean distance in the original algorithm. Perform dynamic neighborhood reconstruction based on communication topology similarity. S37. By comparing the reconstructed evolutionary neighborhood, the dynamic penalty term is calculated. Combined with the affinity matching score, the target node set is output.

5. The computing power scheduling method for AI server clusters based on big data mining according to claim 4, characterized in that, Specifically, S37 includes: S371. In the reconstructed evolutionary neighborhood, select neighboring individuals and perform crossover mutation operations with the current individual to generate offspring individuals. That is, generate a random number between 0 and 1 as the crossover probability. When the random number is less than the preset crossover probability, disconnect the node allocation sequence of the current individual and the neighboring individuals at the preset crossover point and exchange the second half of the disconnected sequence to splice it into a new sequence. S372. Then, a random number between 0 and 1 is generated as the mutation probability. When the random number is less than the preset mutation probability, a position is randomly selected in the node allocation sequence of the offspring individual, and the node assigned to the position is replaced with another node randomly selected in the cluster. The inter-node communication bandwidth margin of the candidate node assigned to the offspring individual is read, and the inter-node communication bandwidth margin is compared with the preset bandwidth threshold to determine whether the inter-node communication bandwidth margin is lower than the preset bandwidth threshold. S373. If the inter-node communication bandwidth margin is lower than the preset bandwidth threshold, an adaptive penalty mechanism is triggered. The preset bandwidth threshold is subtracted from the inter-node communication bandwidth margin to obtain the current bandwidth deficit. The current iteration number and the preset total iteration number are read. The current iteration number is divided by the preset total iteration number to obtain the iteration progress ratio. The current bandwidth deficit is multiplied by the iteration progress ratio to calculate the dynamic penalty term. S374. Perform an addition operation on the dynamic penalty term and the output value of the Chebyshev aggregation function, and use the sum as the weighted affinity matching score. Since the smaller the affinity matching score, the higher the node matching priority, the larger the affinity matching score after the dynamic penalty term is added, it means that the node has been penalized and weighted down. Compare the weighted affinity matching score of the offspring with the affinity matching score of the individuals in the evolutionary neighborhood. S375. If the affinity matching score of the offspring individual after weight reduction is less than the affinity matching score of the individual in the evolutionary neighborhood, then replace the individual in the evolutionary neighborhood with the offspring individual, and search for co-evolution along the reconstructed neighborhood until the preset termination condition is reached, and output the target node set.

6. The computing power scheduling method for AI server clusters based on big data mining according to claim 1, characterized in that, S4 specifically includes: S41. For the target node set, read the completion time weight and estimated total duration of the AI ​​task to be scheduled. The AI ​​task to be scheduled with a completion time weight greater than the preset time threshold and an estimated total duration less than the preset short task duration threshold is identified as a high-priority burst task. The task with a completion time weight less than or equal to the preset time threshold and an estimated total duration greater than or equal to the preset long task duration threshold is identified as a low-priority long task. S42. Add the peak computing power and minimum memory requirements of high-priority burst tasks on each target node to the current computing power and memory usage of low-priority long tasks running on the target node to obtain the total computing power requirement and the total memory requirement. S43. Subtract the total computing power demand from the total computing power limit of the target node to obtain the computing power overflow. Subtract the total video memory demand from the total video memory limit of the target node to obtain the video memory overflow. If the computing power overflow or video memory overflow is greater than zero, it is determined that there is a resource conflict between high-priority burst tasks and low-priority long tasks. S44. If there is a resource conflict, a computing power transfer instruction is issued to the low-priority long task, reducing the current computing precision of the low-priority long task from the preset initial precision to the preset transfer precision, and compressing the current batch size of the low-priority long task from the preset initial batch to the preset transfer batch. S45. Based on the preset transfer precision after the reduction and the compressed transfer batch, calculate the computing power capacity and video memory capacity released by low-priority long tasks, and allocate the released computing power capacity and video memory capacity for high-priority burst tasks to occupy. S46. Bind the task identifier of high-priority burst tasks with the task identifier of low-priority long tasks to form a computing power borrowing relationship. Perform a multiplication operation between the estimated execution time of high-priority burst tasks and the preset compensation multiplier to obtain an additional computing power compensation time. Combine the computing power borrowing relationship and the computing power compensation time as a compensation strategy. Summarize the computing power borrowing relationship and the compensation strategy to output resource conflict transfer and compensation records.

7. The computing power scheduling method for AI server clusters based on big data mining according to claim 1, characterized in that, S5 specifically includes: S51. After the high-priority burst task is completed, read the computing power borrowing relationship in the resource conflict transfer and compensation record, and locate the corresponding low-priority long task according to the computing power borrowing relationship. S52. Read the compensation strategy in the resource conflict transfer and compensation record, extract the computing power compensation duration in the compensation strategy, extend the current scheduling deadline of the low-priority long task by the value of the computing power compensation duration, and return the extended time period as an additional computing power compensation duration to the low-priority long task. S53. Restore the computational precision of the low-priority long task from the preset surrender precision to the preset initial precision, restore the compressed batch size from the preset surrender batch to the preset initial batch, and return the released computing power and video memory capacity to the low-priority long task. S54. Extract all resource conflict judgment results of high-priority burst tasks and low-priority long tasks and integrate them with computing power transfer execution records into conflict handling results. Extract the affinity matching score output by the Chebyshev aggregation function as the affinity matching score filtering result. S55. Align and concatenate the conflict resolution results and affinity matching score filtering results according to the task identifier. Generate a scheduling data table by extending the deadline of low-priority long tasks, restoring the accuracy, and batch overwriting the original task parameters, and use it as the final computing power scheduling decision output.

8. The computing power scheduling method for AI server clusters based on big data mining according to claim 1, characterized in that, S6 specifically includes: S61. Issue the final computing power scheduling decision to each node of the AI ​​server cluster to execute task deployment. At the same time, read the GPU computing utilization and video memory usage of each node according to the preset monitoring cycle. Mark the nodes with GPU computing utilization below the preset computing drop threshold or the nodes with video memory usage suddenly dropping to zero as fault nodes. S62. When a faulty node is detected, extract the tasks running on the faulty node as tasks to be migrated. In the AI ​​server cluster, among the remaining nodes other than the faulty node, select nodes whose inter-node communication bandwidth margin is greater than the communication density identifier of the task to be migrated and whose GPU computing utilization and video memory usage are both lower than the preset migration load threshold. Select the node with the smallest affinity matching score from the selected nodes as the takeover node. S63. Allocate computing and storage resources on the takeover node and start the process of the task to be migrated. At the same time, copy and transmit the unfinished computing data and model parameter files of the task to be migrated on the faulty node to the takeover node through the network channel. During the data transmission, the faulty node continues to execute the untransmitted part of the computing task until the data transmission volume of the takeover node and the remaining computing volume of the faulty node are completed synchronously. S64. Calculate the additional computing power loss time and network transmission time caused by node failure and task migration. Perform an addition operation on the additional computing power loss time and network transmission time to obtain the scheduling overhead. Combine the scheduling overhead, the node identifier of the faulty node and the node identifier of the takeover node to generate scheduling closed-loop feedback data. S65. Read the scheduling overhead in the scheduling closed-loop feedback data. When the scheduling overhead is greater than the preset overhead warning threshold, add the preset weight correction step size to the values ​​of computing power weight and memory weight in the preset resource weights. Subtract twice the preset weight correction step size from the value of communication weight to adaptively correct the hunger weight. At the same time, add the preset threshold correction step size to the value of the preset bandwidth threshold to adaptively correct the affinity matching penalty coefficient.

Images