Dynamic data scheduling method and system for computing power cluster

Abstract

The invention discloses a dynamic data scheduling method and system of a computing power cluster, and belongs to the field of computer science and technical field, and the method comprises the following steps: reading node resources of the computing power cluster, synchronously obtaining a task scale, initializing a dynamic granularity distribution network, and determining splitting granularity; task splitting is carried out according to the splitting granularity, and task priorities are configured; performing load and task execution end prediction on each node in the computing power cluster through the prediction model; performing resource adaptation optimization of the M sub-tasks according to the prediction result; establishing a dynamic optimization channel, performing dynamic optimization on an initial allocation result, and performing task preemption optimization based on task priorities; and performing dynamic data scheduling management of the computing power cluster. According to the method and the device, the technical problems of resource waste, low task execution efficiency and slow system response caused by inflexible resource scheduling and unreasonable task allocation of the computing power cluster under the condition of dynamic load and task scale change in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer science and technology, and in particular to a dynamic data scheduling method and system for a computing power cluster. Background Art

[0002] With the rapid development of cloud computing, big data, artificial intelligence and other technologies, computing clusters have become the key infrastructure to support modern computing-intensive applications. By connecting multiple computing nodes, computing clusters can achieve resource sharing and parallel processing of tasks, providing powerful computing capabilities to meet the needs of large-scale data processing, deep learning, scientific computing and other needs.

[0003] Currently, most computing clusters still face many challenges in the resource scheduling process. Traditional static scheduling methods often ignore the dynamic changes in the task execution process, resulting in waste of computing resources or unbalanced scheduling. Although dynamic scheduling methods can be adjusted according to real-time load, they often rely on relatively simple algorithms and lack refined task granularity division and resource optimization mechanisms. Therefore, how to reasonably schedule tasks and optimize resources through intelligent methods in a dynamic environment has become the key to improving the performance of computing clusters. Summary of the invention

[0004] The present application provides a dynamic data scheduling method and system for a computing power cluster, aiming to solve the technical problems in the prior art that, when the computing power cluster is subjected to dynamic load and task scale changes, resource scheduling is inflexible and task allocation is unreasonable, resulting in resource waste, low task execution efficiency and slow system response.

[0005] In view of the above problems, the present application provides a dynamic data scheduling method and system for a computing power cluster.

[0006] The first aspect disclosed in the present application provides a dynamic data scheduling method for a computing power cluster, the method comprising reading the node resources of the computing power cluster, and synchronously obtaining the task scale, performing dynamic granularity allocation according to the node resources and the task scale; after network initialization, determining the split granularity; performing task splitting according to the split granularity, establishing M subtasks, and configuring the task priorities of the M subtasks; predicting the load and task execution end of each node in the computing power cluster through a prediction model, and establishing a prediction result; performing resource adaptation and optimization for the M subtasks according to the prediction result, and establishing an initial allocation result; establishing a dynamic optimization channel, dynamically optimizing the initial allocation result through the dynamic optimization channel, and establishing a dynamic optimization result, wherein the dynamic optimization includes task preemption optimization based on the task priority; and performing dynamic data scheduling management of the computing power cluster according to the dynamic optimization result.

[0007] Another aspect disclosed in the present application provides a dynamic data scheduling system for a computing power cluster, the system including a split granularity determination module, which is used to read the node resources of the computing power cluster and synchronously obtain the task scale, and determine the split granularity after the dynamic granularity allocation network is initialized according to the node resources and the task scale; a priority configuration module, which is used to split the task according to the split granularity, establish M subtasks, and configure the task priorities of the M subtasks; a prediction result establishment module, which is used to predict the load and task execution end of each node in the computing power cluster through a prediction model, and establish a prediction result; an initial allocation result establishment module, which is used to perform resource adaptation and optimization for the M subtasks according to the prediction result, and establish an initial allocation result; an optimization result establishment module, which is used to establish a dynamic optimization channel, and dynamically optimize the initial allocation result through the dynamic optimization channel to establish a dynamic optimization result, wherein the dynamic optimization includes task preemption optimization based on the task priority; a data scheduling management module, which is used to perform dynamic data scheduling management of the computing power cluster according to the dynamic optimization result.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0009] By adopting a technical solution based on dynamic granularity allocation and task splitting, the granularity of tasks, node resource allocation and priority management are adjusted in real time, solving the technical problems in the existing technology that computing clusters are inflexible in resource scheduling and unreasonable in task allocation under dynamic load and task scale changes, resulting in resource waste, low task execution efficiency and slow system response. This solution can be flexibly adjusted according to the actual situation of tasks and resources, improve resource utilization, reduce task execution delays, and optimize overall scheduling efficiency, thereby achieving the technical effect of improving the processing capacity of computing clusters, reducing resource waste, and improving system response speed and task execution efficiency.

[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of a flow chart of a dynamic data scheduling method for a computing power cluster is provided for an embodiment of the present application;

[0012] Figure 2 A structural diagram of a dynamic data scheduling system for a computing power cluster is provided for an embodiment of the present application.

[0013] Explanation of the reference numerals: splitting granularity determination module 11 , priority configuration module 12 , prediction result establishment module 13 , initial allocation result establishment module 14 , optimization result establishment module 15 , data scheduling management module 16 . DETAILED DESCRIPTION

[0014] The overall idea of ​​the technical solution provided by this application is as follows:

[0015] The embodiment of the present application provides a dynamic data scheduling method and system for a computing power cluster. First, by reading the node resources and task scale, the split granularity is determined and the task is split. Then, the prediction model is used to analyze the node load and task execution status to generate resource adaptation results. Finally, the initial allocation results are adjusted through the dynamic optimization channel, combined with factors such as task priority, load balancing and parallel optimization, to ensure efficient use of resources, improve task execution efficiency, reduce delays, and improve the overall performance of the system.

[0016] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application will be specifically described below in conjunction with the drawings in the specification.

[0017] Embodiment 1, as Figure 1 As shown, an embodiment of the present application provides a dynamic data scheduling method for a computing power cluster, the method comprising:

[0018] Step S100: Read the node resources of the computing power cluster and synchronously obtain the task scale, perform dynamic granularity allocation network initialization according to the node resources and the task scale, and then determine the split granularity.

[0019] Specifically, a computing power cluster is a computing resource pool composed of multiple computing nodes (such as servers, virtual machines, etc.), which are connected together through a high-speed network to jointly complete large-scale computing tasks. Each node usually has independent CPU, memory, storage and other resources, and can work alone or collaboratively process tasks. Node resources refer to the hardware resources possessed by each computing node in the cluster, such as the number of CPU cores, memory size, storage space, network bandwidth, etc. The configuration of these resources directly affects the ability and efficiency of the node to perform tasks. Task scale refers to the amount of calculation, data volume, required resources, etc. of the task to be performed. When scheduling tasks, task scale is an important basis for determining how to allocate resources and split tasks. Dynamic granular allocation refers to the flexible adjustment of the fine granularity of task allocation according to the actual needs of the task and the availability of resources. Split granularity refers to the granularity used when splitting tasks, that is, the size of the task into several subtasks. If the granularity is too large, it may lead to uneven resource allocation; if the granularity is too small, it may increase the complexity of management. By dynamically adjusting the granularity, cluster resources can be used more effectively. Dynamic granularity allocation network is a model or algorithm framework used to dynamically adjust the granularity of task allocation according to task scale and node resources.

[0020] First, you need to read the resource information of each node in the computing power cluster and synchronously obtain the scale of the task. This information is the basis for subsequent resource scheduling. For a task, the task scale may include the computing resources required for the task, the amount of data, the parallelism of the task, etc. Node resources include the number of CPU cores, memory size, storage space, etc. of the computing node.

[0021] Next, based on this resource information and task size, a dynamic granularity allocation strategy is used to determine how to allocate tasks to different computing nodes. At this point, the network is initialized to prepare for scheduling, which usually means establishing a network topology to ensure that the connection between each node is unobstructed so that subsequent tasks can communicate efficiently. Then, the task is split into multiple subtasks according to the split granularity. For example, if the task scale is large and the node resources are large, the split granularity may choose a finer granularity, that is, split the task into multiple smaller subtasks and assign them to different nodes for parallel processing; if the task scale is small and the node resources are tight, a coarser split granularity may be used to merge multiple small tasks on one node for processing. This can effectively avoid resource waste and improve task execution efficiency.

[0022] Through resource scheduling and task splitting strategies, the computing cluster can dynamically adjust resource allocation according to the actual load and task scale, which not only improves the efficiency of cluster resource utilization, but also reduces the time and resource waste of task scheduling, and improves overall computing performance.

[0023] Step S200: split the task according to the split granularity, establish M subtasks, and configure the task priorities of the M subtasks.

[0024] Specifically, the task priority indicates the urgency or importance of task execution, and is used by the scheduling system to decide which tasks to execute first when resources are limited. M subtasks means that the number of subtasks after splitting is M, where M is a dynamically determined value and is a positive integer greater than or equal to 1, depending on the split granularity and task scale.

[0025] First, the large task is decomposed into M subtasks according to the determined splitting granularity. The task splitting process usually uses task scheduling tools, such as big data processing frameworks such as Apache Spark or Hadoop, which support efficient task splitting and parallel execution. Each subtask after splitting needs to be configured with a task priority, and the steps involve evaluating the importance and urgency of each subtask and its impact on the overall task completion time. Priority configuration can be achieved through predefined rules. For example, scheduling algorithms (such as priority scheduling, round-robin scheduling, etc.) can be used to automatically assign priorities based on historical data and task characteristics.

[0026] By splitting tasks according to the granularity and configuring the priority of subtasks, the computing cluster can achieve more efficient resource utilization and task execution. It can reasonably allocate computing resources according to the importance and urgency of the tasks, ensure that key tasks are completed first, and optimize the execution time of the overall tasks.

[0027] Step S300: Use the prediction model to predict the load and task execution completion of each node in the computing power cluster and establish a prediction result.

[0028] Specifically, a prediction model is a mathematical model built through machine learning or statistical methods based on historical data and feature information, and is used to predict future events or states. In a computing power cluster, a prediction model is usually used to predict the load of a node, the execution progress of a task, etc. The prediction result refers to the predicted value of a future state or event calculated by the prediction model. In a computing power cluster, the prediction result includes information such as the load of a node and the time it takes to complete a task.

[0029] First, a training dataset is established by collecting historical task scheduling data, including the computing, storage and network resource usage of each node, the execution time of each task, etc. Then, a regression model is used for training, so that the model can predict future load and execution end time based on node resources and task characteristics.

[0030] In actual use, deep learning models (such as long short-term memory networks (LSTM)) can be used to process time series data and predict the dynamic change trend of node load. Through these prediction results, the scheduling system can dynamically adjust the distribution of tasks, avoid overloaded nodes, complete task scheduling in advance, and thus optimize the overall efficiency of the computing cluster.

[0031] By using the prediction model to predict the load and task execution end time, the utilization rate of computing cluster resources can be significantly improved, avoiding the waste or overload of node resources. In addition, predicting the task completion time can help the system prepare the next stage of tasks before the task execution is completed, reducing waiting time and further improving the overall throughput of the computing cluster.

[0032] Step S400: performing resource adaptation optimization for the M subtasks according to the prediction results, and establishing an initial allocation result.

[0033] Specifically, resource adaptation refers to dynamically adjusting the resource allocation method according to the needs of tasks and nodes to achieve optimal resource utilization. Ensure that tasks can obtain sufficient computing, storage, and network resources during execution, thereby improving task execution efficiency and the overall performance of the computing cluster. The initial allocation result refers to the first resource allocation plan based on the prediction results, according to information such as task scale, node resources, and task priority. It is usually not the final plan and needs to be dynamically optimized.

[0034] The prediction results can be used to understand the current load and idle resources of each node, and appropriate node resources can be allocated to each subtask according to the size and priority of the task. For example, if the load of a node is low and it is predicted that the node will be idle in the future, more subtasks can be allocated to the node to optimize resource utilization.

[0035] Use heuristic algorithms (such as genetic algorithms, particle swarm optimization, etc.) or optimization methods such as linear programming to further adjust the resource allocation of each subtask and seek the optimal resource match. The optimization goal is usually to minimize the completion time of the task or maximize the utilization of resources. After completing resource adaptation, an initial task allocation plan is obtained. These allocation results are dynamically adjusted according to the computing power of the node, the requirements of the task, and the prediction results, so as to achieve effective scheduling of tasks and rational use of resources.

[0036] By dynamically adjusting resource allocation, we ensure that each subtask can be executed on the appropriate node, thereby optimizing task execution efficiency and resource utilization of the computing cluster. Through prediction results and resource adaptation optimization, the system can effectively reduce task waiting time, improve overall computing performance, avoid computing bottlenecks, and maximize resource utilization. During the peak load period of the cluster, this dynamic allocation can significantly reduce the delay of task execution and improve the responsiveness and stability of the system.

[0037] Step S500: establishing a dynamic optimization channel, dynamically optimizing the initial allocation result through the dynamic optimization channel, and establishing a dynamic optimization result, wherein the dynamic optimization includes performing task preemption optimization based on the task priority.

[0038] Specifically, the dynamic optimization channel is a mechanism for real-time adjustment and optimization of resource allocation plans. In a computing power cluster, the initial allocation plan can be continuously adjusted based on information such as the real-time load of the system, task execution status, and node resource utilization to ensure optimal utilization of cluster resources. Task preemption optimization refers to allowing high-priority tasks to preempt the resources of low-priority tasks when resources are insufficient or task priorities change. The dynamic optimization result refers to the resource allocation plan that is finally determined after multiple adjustments, which can most effectively utilize the resources of the computing power cluster and ensure that tasks are completed on time and efficiently.

[0039] The dynamic optimization channel is essentially a real-time feedback mechanism that makes dynamic adjustments based on the actual status of node resources in the cluster, task execution progress and priority, etc. It continuously monitors resource usage and makes decisions based on task priority, load information, etc. By building a dynamic optimization channel, you can respond immediately to any resource bottlenecks or task delays during task execution.

[0040] Through this channel, the initial allocation results are input into the optimization algorithm, which will analyze the execution status and resource consumption of each subtask in real time and adjust the resource allocation of each subtask. For example, if the load of a node is too high, the optimization channel will adjust the resource allocation of subtasks or migrate tasks to other idle nodes to ensure the smooth operation of the system. If it is found during the execution process that the resource requirements of a high-priority task are not met in time, the dynamic optimization channel will start the task preemption mechanism.

[0041] Through dynamic optimization channels, the computing cluster can continuously adjust resource allocation plans according to factors such as load changes and task priorities during actual operation, improve resource utilization, and reduce task waiting time. During task execution, dynamic optimization can not only improve the completion efficiency of high-priority tasks, but also ensure that low-priority tasks are completed smoothly when resources are sufficient.

[0042] Step S600: Perform dynamic data scheduling management of the computing power cluster according to the dynamic optimization result.

[0043] Specifically, dynamic data scheduling management refers to continuously adjusting and optimizing task scheduling strategies based on real-time collected computing cluster data to improve computing efficiency and resource utilization. This management method can not only adapt to changes in cluster status, but also dynamically allocate resources according to different task requirements to ensure that tasks are completed on time.

[0044] First, through the dynamic optimization channel, combined with the analysis results of multiple sub-channels such as task preemption optimization, load balancing optimization and parallel collaborative optimization, a final dynamic optimization result is obtained. This result reflects the optimal resource allocation strategy, that is, how to reasonably allocate tasks to different nodes, adjust priorities, coordinate parallel execution, etc.

[0045] According to the dynamic optimization results, the task scheduling management in the computing power cluster is started. The system monitors the resource utilization of each node in real time, including computing load, storage and network bandwidth. When the load of a node is too high, the system will automatically migrate some tasks to nodes with lower load to achieve load balancing.

[0046] During the task execution process, the scheduling management system will dynamically adjust resource allocation based on the real-time progress of the task and the resource usage of the node. For example, if a high-priority task executes slowly, the scheduling system will immediately preempt resources and allocate more computing resources to the task to ensure that the task can be completed on time.

[0047] For tasks that can be executed in parallel, the system will split the task into multiple subtasks to ensure that they can be executed in parallel on different nodes, thereby shortening the overall execution time. According to the computing requirements of each subtask and the load of the node, the allocation and scheduling of subtasks are adjusted to ensure maximum parallel efficiency.

[0048] By implementing dynamic data scheduling management, the computing cluster can adjust resource allocation according to real-time load and task requirements, optimizing overall resource utilization and task completion time. Especially in the case of uneven load and changing task priorities, dynamic scheduling can ensure that high-priority tasks are executed first, avoiding resource waste and task delays. This dynamic scheduling management method improves the system's flexibility, response speed, and computing efficiency, and can adapt to changing computing needs.

[0049] Furthermore, the dynamic optimization of the initial allocation result is performed through the dynamic optimization channel to establish a dynamic optimization result, including: activating a priority comparison sub-channel in the dynamic optimization channel, and using the priority comparison sub-channel to perform a subtask priority comparison based on the initial allocation result, the subtask priority comparison being a priority comparison of the subtasks under the execution of the subtasks and the initial allocation result; establishing a priority exception, the priority exception being a comparison result in which the priority comparison result meets a preset threshold, performing a migration cost calculation based on the priority exception, and establishing a migration cost calculation result; performing an optimization judgment based on the migration cost calculation result and the priority result of the high-priority subtask, and if the optimization judgment is a passing result, performing the task preemption optimization of the priority exception.

[0050] Specifically, the priority comparison subchannel is a submodule in the dynamic optimization channel, which is responsible for comparing the priorities of task subtasks. Its function is to compare the priority of the currently assigned task with the priority in the initial allocation plan to determine whether the execution order of the task needs to be adjusted. Subtask priority comparison refers to comparing the priorities of each subtask, in order to ensure that during the task scheduling process, subtasks with higher priorities can obtain resources first, thereby avoiding blocking or delaying the execution of important tasks by low-priority tasks. Priority anomalies refer to situations where the priorities found through priority comparison are inconsistent or do not comply with the preset strategy. Migration cost refers to the cost of migrating certain subtasks from one node to another in order to optimize resource allocation. This cost can be time delay, data transmission overhead, resource consumption, etc. The higher the migration cost, the greater the cost of the migration operation.

[0051] First, activate the priority comparison subchannel in the dynamic optimization channel. Compare the priorities of the subtasks currently being executed with those in the initial allocation results, i.e., subtask priority comparison. Once a priority anomaly is detected, the system will perform a migration cost calculation to evaluate the cost of migrating low-priority tasks to other nodes or reallocating resources. Use optimization algorithms, such as linear programming or heuristic algorithms, to evaluate the cost-effectiveness of different migration schemes. The migration cost calculation results will be used for optimization judgment, i.e., to determine whether it is worthwhile to perform task preemption optimization.

[0052] If the optimization judgment result is passed, the system will perform task preemption optimization, that is, transfer resources from low-priority tasks to high-priority tasks. For example, in a data analysis cluster, real-time data processing tasks (high priority) need to obtain computing resources first, while background data backup tasks (low priority) can be temporarily migrated to other nodes with sufficient resources to ensure the timely completion of real-time tasks.

[0053] Through this step, the computing power cluster can achieve more refined and dynamic resource management, ensuring that high-priority tasks can obtain the required resources first when resources are tight, thereby improving the overall task execution efficiency and system responsiveness.

[0054] Furthermore, the method also includes: activating the load balancing sub-channel within the dynamic optimization channel, using the load balancing sub-channel to perform node load balancing analysis based on the initial allocation result, and establishing a balanced adjustment optimization plan; activating the parallel subtask optimization channel within the dynamic optimization channel, using the parallel subtask optimization channel to perform parallel collaborative analysis of subtasks of the initial allocation result, and establishing a parallel collaborative adjustment optimization plan; jointly analyzing the task preemption optimization, balanced adjustment optimization plan, and collaborative adjustment optimization plan to establish a dynamic optimization result.

[0055] Specifically, the load balancing sub-channel is a sub-module in the dynamic optimization channel, which is specifically used to analyze the load of each node in the computing power cluster. The main goal is to reasonably allocate the computing and storage resources of each node by adjusting resource allocation or task scheduling, avoid overloading or idleness of some nodes, and ensure load balancing of the entire cluster. The parallel subtask optimization channel is a sub-module in the dynamic optimization channel, which is specifically used to analyze the parallelism between subtasks after task splitting. The goal is to optimize the scheduling and collaborative execution of subtasks so that tasks can be executed in parallel more efficiently and the total execution time can be shortened.

[0056] First, the load balancing subchannel is activated and begins to analyze the resource usage of each node in the cluster. Specifically, load balancing algorithms (such as weighted round-robin, least connections, etc.) are used to analyze and adjust node loads. Assuming that the CPU and memory usage of a node is close to the upper limit, while the resources of other nodes are idle, the load balancing subchannel will propose a balancing adjustment optimization plan, such as migrating some tasks from high-load nodes to idle nodes to ensure load balancing of all nodes.

[0057] Next, the parallel subtask optimization channel will be activated to perform subtask parallel collaborative analysis on the initial task allocation results. Specifically, a task scheduling optimization algorithm (such as graph coloring algorithm, genetic algorithm, deep learning, etc.) is used to analyze the parallelism and collaborative execution between subtasks. For example, a task is split into multiple subtasks, some of which can be executed in parallel, while others have data dependencies. In this case, the parallel subtask optimization channel will analyze the dependencies between subtasks and propose optimization solutions, including adjusting the execution order of subtasks or changing the degree of parallelism to improve the overall execution efficiency of the task.

[0058] Finally, all optimization schemes (task preemption optimization, load balancing optimization, and parallel collaborative optimization) will be comprehensively evaluated through joint scheme analysis. The purpose of this analysis is to integrate the advantages of each optimization scheme and formulate an optimal resource allocation scheme. For example, some high-priority tasks need to preempt resources, while load balancing optimization ensures that resources are not overly concentrated on a few nodes, and parallel collaborative optimization ensures efficient parallel execution of tasks.

[0059] By combining the three optimization schemes, we can dynamically adjust resource allocation to ensure that tasks are completed efficiently in the shortest possible time, improve cluster resource utilization and task execution efficiency. This significantly improves the processing capabilities of large-scale computing tasks in a dynamic environment, and is particularly suitable for computing scenarios that require high concurrency and high efficiency.

[0060] Furthermore, after the dynamic granularity allocation network is initialized according to the node resources and the task scale, the split granularity is determined, including: utilizing the node feature extraction subnetwork of the dynamic granularity allocation network to extract features of the node resources, establishing node quantity features and node balanced computing power features, and establishing a first split granularity constraint based on the node quantity features and the node balanced computing power features; utilizing the task feature extraction subnetwork of the dynamic granularity allocation network to extract features of the task scale, establishing task topology structure features and task computing requirement features, and establishing a second split granularity constraint based on the task topology structure features and the task computing requirement features; determining the split granularity based on the first split granularity constraint and the second split granularity constraint.

[0061] Specifically, the node quantity feature refers to the number of nodes involved in the calculation in the cluster. Different numbers of nodes will affect the allocation of resources, and thus affect the granularity of task splitting. The node balanced computing power feature refers to the distribution of computing power among the nodes in the cluster. If the computing power of some nodes is significantly stronger than that of other nodes, tasks will tend to be assigned to nodes with strong resources, while weaker nodes will take on smaller tasks. The first split granularity constraint sets preliminary constraints for task splitting based on the node quantity feature and the node balanced computing power feature. The second split granularity constraint sets further constraints for task splitting based on the task topology structure feature and the task computing requirement feature.

[0062] First, the node feature extraction subnetwork in the dynamic granularity allocation network is used to perform detailed analysis and feature extraction of the resources of each node in the computing power cluster. This includes extracting node quantity features and node balanced computing power features. These features help the system understand the resource configuration and load status of the current cluster. Specifically, a dynamic granularity allocation network is obtained, and a neural network model is constructed using a deep learning method. First, a data set containing node resource and task scale features is collected and prepared as training input. Then, a network architecture is designed, including a node feature extraction subnetwork and a task feature extraction subnetwork, which process different types of input data respectively. Using a supervised learning method, the model is trained to optimize network parameters by using the labeled optimal split granularity as the output. During the training process, cross-validation and hyperparameter tuning can be used to ensure that the model has good generalization capabilities. Finally, the trained network is deployed for real-time dynamic granularity allocation.

[0063] Next, the task feature extraction subnetwork is used to extract features of the scale of the scheduled tasks, and obtain the task topology structure features and task computing requirement features. Through these features, the system can fully understand the complexity and resource requirements of the task. After extracting the features of the nodes and tasks, based on the first split granularity constraint and the second split granularity constraint, the most appropriate split granularity is determined by comprehensively considering the distribution of node resources and the dependencies of tasks. For example, for a computationally intensive task, if the CPU utilization of most nodes in the cluster is low and the memory is sufficient, a finer split granularity will be selected to split the task into more subtasks and assign them to different nodes for parallel processing. On the contrary, if the tasks are highly dependent and some nodes in the cluster are resource-constrained, a coarser split granularity will be selected to merge multiple subtasks on the same node to reduce communication overhead and resource competition.

[0064] Through this step, the computing power cluster can intelligently determine the task splitting granularity based on the characteristics of node resources and task scale, and achieve efficient resource allocation and task scheduling.

[0065] Furthermore, the dynamic data scheduling management of the computing power cluster based on the dynamic optimization result includes: backtracing the task execution of the nodes in the computing power cluster and establishing a task backtracing window; performing execution adaptation of the nodes and tasks based on the task backtracing window and establishing node execution taboo values; performing taboo adaptation optimization of the dynamic optimization result through the node execution taboo value, and performing dynamic data scheduling management of the computing power cluster based on the taboo adaptation optimization result.

[0066] Specifically, the task execution backtracking of a node refers to analyzing and tracing the task execution history of each computing node in the computing power cluster to evaluate the performance, efficiency, and possible problems of the node in the task execution process. The task backtracking window refers to the time window used in the backtracking analysis, which includes the task execution data within a period of time in the node execution history. The node execution taboo value is a threshold or limit obtained by analyzing the backtracking data of the node's historical task execution, indicating that the node has a performance bottleneck or abnormal state when executing certain tasks.

[0067] First, based on historical data, the task execution of each computing node is retrospectively analyzed. By building a task retrospective window, the system can view the task execution status of the node in the past period of time, such as key indicators such as load changes and task delays. The task data in this window will be used for subsequent analysis. For example, if a node frequently fails or delays in task execution in the past period of time, the system can identify potential problems with the node.

[0068] Next, based on the results of the backtracking analysis, the system will optimize the adaptation relationship between nodes and tasks. The historical data in the task backtracking window reveals which tasks performed poorly when running on a specific node. Based on this historical data, the system establishes a "taboo value" for each node, that is, which task types or task scales lead to poor execution results for this node. For example, if a node has performed poorly in the past when executing large-scale computing-intensive tasks, the system will set the taboo value of the node to "cannot execute large-scale computing tasks."

[0069] When performing taboo adaptation optimization, the system adjusts the initial dynamic optimization results according to the taboo value to avoid inappropriate task scheduling. For example, if the initial allocation result of a task shows that it is assigned to a node with a high load, and the taboo value of the node indicates that it cannot carry the high-load task, the system will reschedule the task to a more suitable node.

[0070] After completing the taboo adaptation optimization, the dynamic data scheduling management of the entire computing power cluster will rely on the optimized task allocation strategy for execution. At this time, the system will ensure that each task runs on the node with the most suitable resources to avoid new resource bottlenecks or delays caused by poor performance in historical data.

[0071] Through this process, the computing power cluster can achieve more intelligent resource scheduling management. Taboo adaptation optimization can effectively avoid resource waste or delays caused by historical performance issues during task execution, and improve the stability and efficiency of task execution.

[0072] Furthermore, the method also includes: monitoring the execution of tasks on the computing power cluster and establishing a monitoring data set; evaluating the execution status of nodes through the monitoring data set and establishing independent adaptation feedback; analyzing the execution status of tasks and establishing global adaptation feedback; generating enhanced incentives using the independent adaptation feedback and the global adaptation feedback, and performing enhanced management of dynamic data scheduling of the computing power cluster based on the enhanced incentives.

[0073] Specifically, the monitoring data set is a collection of data related to tasks and nodes collected through execution monitoring. It usually includes information such as node load, resource usage, task execution time, task priority, etc., which is used to further analyze and optimize the scheduling strategy. Independent adaptation feedback is an independent evaluation result for each node or task, which aims to feedback optimization measures based on the execution of a single node or task. For example, when a node's task is executed too slowly, the system will adjust the resource allocation of the node based on the feedback. Global adaptation feedback is feedback after evaluating the overall execution status of all tasks and nodes in the entire computing power cluster. It can identify the overall performance bottleneck in the cluster and provide a global resource optimization solution. Enhanced incentives are incentive mechanisms generated based on independent adaptation feedback and global adaptation feedback for further optimizing scheduling strategies. Enhanced incentives guide the system to optimize in a more efficient direction by adjusting the scheduling strategy.

[0074] First, by real-time monitoring of the task execution process of each node in the computing power cluster, the system will collect information such as the node load, task execution progress, execution time, resource consumption, etc. This information is organized into a monitoring data set as the basis for subsequent optimization. Using the monitoring data set, the execution status of each node is analyzed to generate a node execution status evaluation. For example, if the task execution delay of a node is too long, independent adaptation feedback will be generated for the node. Feedback measures include migrating tasks to nodes with lower load or adjusting the resource configuration of the node. This feedback helps optimize node resource allocation and improve the execution efficiency of each node.

[0075] By globally analyzing the task execution status of the entire cluster, the system evaluates the overall load of the cluster, the coordination of task execution, and the balanced utilization of resources. For example, if some nodes are found to be overloaded while other nodes are idle, the system will adjust the cluster's resource allocation through global adaptive feedback to improve overall performance.

[0076] Based on independent adaptation feedback and global adaptation feedback, the system generates enhanced incentives. For example, if a task has a high priority but is delayed due to insufficient node resources, enhanced incentives include increasing the priority of the task, allocating more resources, or even reserving resources in advance. Finally, based on these incentives, the system performs dynamic data scheduling enhancement management on the cluster to further improve the intelligence of scheduling and the efficiency of execution.

[0077] Through this method, the computing cluster can monitor and optimize task execution in real time, improve resource utilization, and avoid the situation where a single node is overloaded or idle. The delay in task execution and resource waste are effectively reduced, and the overall computing efficiency and response speed of the cluster are significantly improved.

[0078] Furthermore, the method also includes: identifying node anomalies based on the monitoring data set to establish an anomaly identification data set; extracting features of anomaly frequency, anomaly duration, and anomaly trigger peak value from the anomaly identification data set, and generating cumulative node anomalies based on the feature extraction results; and using the cumulative node anomalies to issue early warning alerts for the computing power cluster.

[0079] Specifically, the anomaly identification dataset is to extract feature data related to node anomalies by screening and analyzing the data in the monitoring dataset. It contains various types of information related to node performance problems, task failures, or resource mismatches, providing a basis for subsequent anomaly analysis and identification. The anomaly frequency refers to the number of times a node exhibits abnormal behavior within a certain time range. A higher anomaly frequency means that the node is in an unstable state, or the cluster is overloaded, resulting in frequent problems. The duration of the anomaly refers to the length of time that the abnormal behavior persists. A longer duration of the anomaly usually means that there are serious problems with the node or task, resulting in task failure or system crash. The anomaly trigger peak refers to the peak value of the relevant system indicators when the anomaly occurs, indicating the severity of the problem. The cumulative node anomaly refers to the accumulation of all abnormal behaviors that occur on the node within a certain time range. It is a quantitative assessment of node stability, which can reflect the performance and risks of the node in long-term operation.

[0080] First, the node is identified for anomalies through the monitoring data set. This process uses machine learning algorithms (such as anomaly detection algorithms) to analyze data such as resource usage and task execution status of each node to detect potential anomalies. For example, if the CPU utilization of a node continues to exceed 90%, it will be considered an abnormal signal. After anomaly identification, the system generates an anomaly identification data set, which contains all events identified as abnormal.

[0081] Next, based on the anomaly identification dataset, the system extracts several key features, including anomaly frequency, anomaly duration, and anomaly triggering peaks. These features help assess the severity of the anomaly. Subsequently, the system generates cumulative node anomalies based on the extracted features, that is, the abnormal behaviors of all nodes in a time period are accumulated and analyzed. This process can identify nodes or tasks with long-term problems and provide a perspective on the overall health status. Finally, based on the cumulative node anomalies, the system will issue a pre-alarm. If the abnormal behavior of a node is frequent, lasts for a long time, and the triggered peak exceeds the preset threshold, the system will automatically issue an alarm.

[0082] Through the abnormal identification and early warning mechanism, the computing power cluster can effectively monitor and identify potential node problems, issue alarms in a timely manner, and help administrators quickly locate problems and repair them. This can not only improve the stability and reliability of the computing power cluster, but also reduce task delays or system crashes caused by node failures or resource bottlenecks.

[0083] In summary, the dynamic data scheduling method for a computing power cluster provided in the embodiment of the present application has the following technical effects:

[0084] 1. Through dynamic granularity allocation and task splitting, tasks can be flexibly allocated according to cluster resources and task scale, improving the resource utilization of the computing cluster, reducing the delay and waiting time of task execution, optimizing task scheduling efficiency, and improving the response speed and processing capacity of the overall system.

[0085] 2. Through task backtracking analysis and taboo value mechanism, unreasonable resource allocation can be effectively identified and avoided, and uneven system load and abnormal task execution can be reduced. This method can dynamically adjust the task scheduling strategy, improve the stability of the system and the reliability of task execution, and reduce the waste of computing resources.

[0086] 3. Based on real-time anomaly identification and frequency analysis, it can promptly discover potential performance bottlenecks or faulty nodes in the computing cluster, issue early warnings, reduce the risk of system failures, and improve the continuity and reliability of task execution, thereby ensuring the long-term stable operation of the cluster.

[0087] Embodiment 2 is based on the same inventive concept as the method for dynamic data scheduling of a computing power cluster in the above embodiment. Figure 2 As shown, an embodiment of the present application provides a dynamic data scheduling system for a computing power cluster, the system comprising:

[0088] A split granularity determination module 11 is used to read the node resources of the computing power cluster and synchronously obtain the task scale, and after the dynamic granularity allocation network is initialized according to the node resources and the task scale, the split granularity is determined; a priority configuration module 12 is used to split the task according to the split granularity, establish M subtasks, and configure the task priority of the M subtasks; a prediction result establishment module 13 is used to predict the load and task execution end of each node in the computing power cluster through a prediction model, and establish a prediction result; an initial allocation result establishment module 14 is used to perform resource adaptation and optimization for the M subtasks according to the prediction result, and establish an initial allocation result; an optimization result establishment module 15 is used to establish a dynamic optimization channel, and dynamically optimize the initial allocation result through the dynamic optimization channel to establish a dynamic optimization result, wherein the dynamic optimization includes task preemption optimization based on the task priority; a data scheduling management module 16 is used to perform dynamic data scheduling management of the computing power cluster according to the dynamic optimization result.

[0089] Furthermore, the optimization result establishment module 15 is also used to perform the following steps: activate the priority comparison sub-channel within the dynamic optimization channel, and use the priority comparison sub-channel to perform sub-task priority comparison based on the initial allocation result, wherein the sub-task priority comparison is a priority comparison of the sub-task under the execution sub-task and the initial allocation result; establish a priority exception, wherein the priority exception is a comparison result in which the priority comparison result meets a preset threshold, and perform migration cost calculation based on the priority exception to establish a migration cost calculation result; perform optimization judgment based on the migration cost calculation result and the priority result of the high-priority sub-task, and if the optimization judgment is a passing result, perform task preemption optimization of the priority exception.

[0090] Furthermore, the dynamic data scheduling system of the computing power cluster is also used to perform the following steps: activate the load balancing sub-channel within the dynamic optimization channel, use the load balancing sub-channel to perform node load balancing analysis based on the initial allocation result, and establish a balanced adjustment optimization plan; activate the parallel subtask optimization channel within the dynamic optimization channel, use the parallel subtask optimization channel to perform parallel collaborative analysis of subtasks of the initial allocation result, and establish a parallel collaborative adjustment optimization plan; conduct a joint analysis of task preemption optimization, balanced adjustment optimization plan, and collaborative adjustment optimization plan to establish a dynamic optimization result.

[0091] Furthermore, the split granularity determination module 11 is also used to perform the following steps: utilizing the node feature extraction subnetwork of the dynamic granularity allocation network to extract features of the node resources, establish node quantity features and node balanced computing power features, and establish a first split granularity constraint based on the node quantity features and the node balanced computing power features; utilizing the task feature extraction subnetwork of the dynamic granularity allocation network to extract features of the task scale, establish task topology structure features and task computing requirement features, and establish a second split granularity constraint based on the task topology structure features and the task computing requirement features; determining the split granularity based on the first split granularity constraint and the second split granularity constraint.

[0092] Furthermore, the data scheduling management module 16 is also used to perform the following steps: perform task execution backtracking of nodes in the computing power cluster and establish a task backtracking window; perform execution adaptation of nodes and tasks based on the task backtracking window and establish node execution taboo values; perform taboo adaptation optimization of the dynamic optimization results through the node execution taboo values, and perform dynamic data scheduling management of the computing power cluster according to the taboo adaptation optimization results.

[0093] Furthermore, the dynamic data scheduling system of a computing power cluster is also used to perform the following steps: monitor the execution of tasks on the computing power cluster and establish a monitoring data set; evaluate the node execution status through the monitoring data set and establish independent adaptation feedback; analyze the execution status of the task and establish global adaptation feedback; generate enhanced incentives using the independent adaptation feedback and the global adaptation feedback, and perform enhanced management of the dynamic data scheduling of the computing power cluster based on the enhanced incentives.

[0094] Furthermore, the dynamic data scheduling system for a computing power cluster is also used to perform the following steps: identify node anomalies based on the monitoring data set and establish an anomaly identification data set; extract features of anomaly frequency, anomaly duration, and anomaly trigger peak value from the anomaly identification data set, and generate cumulative node anomalies based on the feature extraction results; and use the cumulative node anomalies to issue early warning alarms for the computing power cluster.

[0095] Any step of the method described above can be stored as a computer instruction or program in an unlimited computer memory, and can be called and recognized by an unlimited computer processor to implement any method in the embodiments of the present application, without any unnecessary restrictions.

[0096] Furthermore, the first or second mentioned above not only represents an order relationship, but also represents a specific concept, and / or refers to the selection of multiple elements individually or in whole. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalent technologies, the present application intends to include these modifications and variations.

Claims

1. A dynamic data scheduling method for a computing power cluster, characterized in that: The method comprises: Read the node resources of the computing power cluster and synchronously obtain the task scale, perform dynamic granularity allocation network initialization according to the node resources and the task scale, and determine the split granularity; Split the task according to the split granularity, establish M subtasks, and configure the task priorities of the M subtasks; Use the prediction model to predict the load and task execution completion of each node in the computing power cluster and establish the prediction results; Perform resource adaptation optimization for the M subtasks according to the prediction results and establish an initial allocation result; Establishing a dynamic optimization channel, dynamically optimizing the initial allocation result through the dynamic optimization channel, and establishing a dynamic optimization result, wherein the dynamic optimization includes performing task preemption optimization based on the task priority; Dynamic data scheduling management of the computing power cluster is performed according to the dynamic optimization results.

2. A dynamic data scheduling method for a computing power cluster as claimed in claim 1, characterized in that: The dynamically optimizing the initial allocation result through the dynamic optimization channel to establish the dynamic optimization result includes: Activate the priority comparison subchannel in the dynamic optimization channel, and use the priority comparison subchannel to perform a subtask priority comparison based on the initial allocation result, wherein the subtask priority comparison is a priority comparison of the subtasks under the execution subtask and the initial allocation result; Establishing a priority exception, wherein the priority exception is a comparison result in which the priority comparison result meets a preset threshold, performing migration cost calculation according to the priority exception, and establishing a migration cost calculation result; An optimization judgment is performed based on the migration cost calculation result and the priority result of the high-priority subtask. If the optimization judgment is a passing result, the task preemption optimization of the abnormal priority is performed.

3. A dynamic data scheduling method for a computing power cluster as claimed in claim 2, characterized in that: The method further comprises: Activate the load balancing subchannel in the dynamic optimization channel, use the load balancing subchannel to perform node load balancing analysis based on the initial allocation result, and establish a balancing adjustment optimization plan; Activate the parallel subtask optimization channel in the dynamic optimization channel, use the parallel subtask optimization channel to perform subtask parallel collaborative analysis of the initial allocation result, and establish a parallel collaborative adjustment optimization plan; Conduct joint analysis on task preemption optimization, balanced adjustment optimization plan, and collaborative adjustment optimization plan to establish dynamic optimization results.

4. The method for dynamic data scheduling of a computing power cluster according to claim 1, characterized in that: After the network is initialized by dynamically allocating granularity according to the node resources and the task scale, the split granularity is determined, including: Utilize the node feature extraction subnetwork of the dynamic granularity allocation network to extract the features of the node resources, establish node quantity features and node balanced computing power features, and establish a first split granularity constraint based on the node quantity features and the node balanced computing power features; Utilize the task feature extraction subnetwork of the dynamic granularity allocation network to extract the features of the task scale, establish task topology features and task computing requirement features, and establish a second split granularity constraint according to the task topology features and the task computing requirement features; The split granularity is determined according to the first split granularity constraint and the second split granularity constraint.

5. The method for dynamic data scheduling of a computing power cluster according to claim 1, characterized in that: The dynamic data scheduling management of the computing power cluster according to the dynamic optimization result includes: Performing task execution backtracking on the nodes of the computing power cluster and establishing a task backtracking window; Performing execution adaptation of nodes and tasks based on the task backtracking window, and establishing node execution taboo values; The taboo value is executed by the node to perform taboo adaptation optimization of the dynamic optimization result, and dynamic data scheduling management of the computing power cluster is performed according to the taboo adaptation optimization result.

6. A method for dynamic data scheduling of a computing power cluster as claimed in claim 1, characterized in that: The method further comprises: Performing task execution monitoring on the computing power cluster and establishing a monitoring data set; Evaluate the node execution status through the monitoring data set and establish independent adaptive feedback; Analyze the execution status of tasks and establish global adaptation feedback; The independent adaptation feedback and the global adaptation feedback are used to generate enhanced incentives, and dynamic data scheduling enhanced management of the computing power cluster is performed based on the enhanced incentives.

7. A method for dynamic data scheduling of a computing power cluster as claimed in claim 6, characterized in that: The method further comprises: Perform node anomaly identification based on the monitoring data set to establish an anomaly identification data set; Extract the abnormal frequency, abnormal duration, and abnormal trigger peak features of the abnormal identification data set, and generate cumulative node abnormalities according to the feature extraction results; The accumulated node anomalies are used to issue early warning alerts for the computing power cluster.

8. A dynamic data scheduling system for a computing power cluster, characterized in that: A method for dynamic data scheduling of a computing power cluster according to any one of claims 1 to 7, the system comprising: A split granularity determination module is used to read the node resources of the computing power cluster and synchronously obtain the task scale, and after the dynamic granularity allocation network is initialized according to the node resources and the task scale, the split granularity is determined; A priority configuration module, used to split the task according to the split granularity, establish M subtasks, and configure the task priorities of the M subtasks; The prediction result establishment module is used to predict the load and task execution completion of each node in the computing power cluster through the prediction model and establish the prediction result; An initial allocation result establishment module is used to perform resource adaptation and optimization of the M subtasks according to the prediction results to establish an initial allocation result; An optimization result establishment module, used for establishing a dynamic optimization channel, dynamically optimizing the initial allocation result through the dynamic optimization channel, and establishing a dynamic optimization result, wherein the dynamic optimization includes performing task preemption optimization based on the task priority; The data scheduling management module is used to perform dynamic data scheduling management of the computing power cluster according to the dynamic optimization results.

Images