Real-time scheduling system and method for cluster computing resources

Abstract

The invention relates to the technical field of resource management, in particular to a cluster computing resource real-time scheduling system and method, and the system comprises a resource fluctuation detection module, a task priority judgment module, a resource allocation conflict recognition module, a task migration track generation module and a scheduling efficiency evaluation module. According to the method, the resource fluctuation time point can be accurately calibrated by collecting the processor and memory utilization rate and analyzing the change trend, the resource change perception capability can be enhanced, the prospective performance of scheduling response can be improved, high-demand tasks are screened in combination with task characteristics, resources are preferentially allocated, the operation timeliness is guaranteed, and the resource conflict state is identified by constructing the conflict index; the adaptive adjustment capability of scheduling is enhanced, a migration path is generated based on a conflict time period, reasonable path selection and load balance acceptance are realized, the resource utilization rate change after migration is analyzed, the migration efficiency is quantified, the allocation strategy is optimized, and the scheduling flexibility, stability and resource utilization efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of resource management, and in particular to a real-time scheduling system and method for cluster computing resources. Background Art

[0002] The field of resource management technology encompasses the configuration, scheduling, monitoring, and optimization of computing resources to ensure efficient and effective execution of computing tasks in complex computing environments. Core areas of this field include the rational allocation of computing resources, the design of task scheduling algorithms, load balancing, and real-time resource monitoring and scheduling optimization. With the rapid development of cloud computing, distributed computing, and big data technologies, the efficient management and scheduling of large amounts of computing resources to meet diverse business needs and improve overall system performance has become a crucial topic in this field. Resource management technology involves not only the management of hardware resources but also the collaborative work of software systems to ensure that computing tasks are completed on time and on demand.

[0003] The real-time scheduling system for cluster computing resources refers to the scheduling and management of resources in a computing cluster through a series of specific means, aiming to solve the resource allocation and task scheduling problems in a cluster computing environment. The topic covers the design and implementation of a real-time scheduling system, mainly through real-time monitoring of resource status, dynamic identification of task requirements, and timely adjustment of scheduling strategies, to ensure that various computing tasks in the system can run efficiently under resource constraints. Specifically, by collecting information such as the operating status and resource load of cluster nodes in real time, combined with the demand characteristics of the task, the corresponding scheduling strategy is adopted to rationally allocate computing resources to maximize the utilization of computing resources in the cluster and optimize task completion time. The system adopts a distributed scheduling method that can dynamically adjust resource allocation according to real-time load changes and task priorities, solving the complexity of resource scheduling in large-scale computing clusters.

[0004] Existing technologies rely on fixed-cycle resource monitoring mechanisms, resulting in an inability to quickly respond to sudden changes in node resource status. Monitoring lags make scheduling decisions lack real-time performance, leading to task delays or resource idleness. In task priority determination, static weights or preset labels are mostly used, ignoring the dynamic resource consumption characteristics of tasks during operation. This can easily lead to inefficient tasks occupying resources for a long time, affecting the scheduling efficiency of critical tasks. Resource conflict identification often remains at the level of simple occupancy ratio judgment, failing to extract conflict characteristics of inter-task competition, making it difficult to cope with complex resource contention. The task migration mechanism lacks comprehensive consideration of the migration path and the resource status of the target node, leading to new performance bottlenecks. The evaluation method is too extensive and lacks dynamic evaluation based on changes in resource utilization, making it difficult to identify efficiency issues in scheduling, affecting the optimization of resource allocation strategies, and making it difficult to maintain scheduling quality during long-term operation. This problem is particularly prominent in large-scale cluster environments, which can easily lead to task accumulation, frequent resource conflicts, and scheduling imbalances, restricting overall performance improvement. Summary of the Invention

[0005] In order to solve the technical problems existing in the prior art, the embodiments of the present invention provide a cluster computing resource real-time scheduling system and method. The technical solution is as follows: In one aspect, a cluster computing resource real-time scheduling system is provided, the system comprising: The resource fluctuation detection module obtains the operating status information of the nodes in the computing cluster, analyzes the changing trends of processor occupancy and memory usage, marks the time points when resource usage shifts, and generates a list of resource fluctuation events; The task priority determination module analyzes the resource demand intensity of the task based on the resource fluctuation event list and the execution characteristic parameters of the task, selects tasks with high demand intensity, and generates a priority queue; The resource allocation conflict identification module extracts the resource competition relationship in the corresponding time period according to the priority queue, determines whether there is a resource allocation conflict state by identifying the resource allocation conflict index, and generates conflict state identification information; The task migration trajectory generation module identifies the task migration path within the conflict time period based on the conflict status identification information, records the node sequence and migration time range of the task migration, and generates task migration trajectory information.

[0006] As a further solution of the present invention, the resource fluctuation event list includes the resource utilization offset time point, fluctuation amplitude, and utilization change trend; the priority queue includes high-demand intensity task identification, task priority sequence number, and task execution characteristic parameters; the conflict status identification information includes resource allocation conflict time period, conflicting task pair, and conflict index; the task migration trajectory information includes migration path, node sequence, and migration time range.

[0007] As a further solution of the present invention, the resource fluctuation detection module includes: The running status acquisition submodule obtains the running status information of the nodes in the computing cluster, including processor occupancy, memory usage, and network bandwidth data. It binds the running status parameters to time tags, establishes a time series data set, and obtains the node running status time series data. The occupancy analysis submodule calls the processor occupancy and memory usage data points in adjacent time periods based on the node operation status time series data, calculates the change rate value between consecutive time points, and performs slope extraction on the change trend curve to generate a resource occupancy change interval set; The fluctuation marking submodule identifies the network bandwidth change rate within the same time period based on the resource occupancy rate change interval set, calculates the deviation value of the two types of change rates in each time period, filters the time nodes where the deviation value exceeds the resource fluctuation threshold, and generates a resource fluctuation event list.

[0008] As a further solution of the present invention, the task priority determination module includes: The demand intensity calculation submodule extracts the task execution characteristics at each time point based on the resource fluctuation event list, including task type, expected execution time, processor resource requirements, and memory resource requirements, and integrates them into data groups according to time series to generate a task characteristic parameter set; The priority screening submodule calculates the demand intensity value of each task based on the task characteristic parameter set, sorts the demand intensity values ​​in time series, marks tasks with a standard deviation higher than the average demand intensity, and generates a demand intensity task set; The queue generation submodule extracts the task identification index and the demand intensity value based on the demand intensity task set, arranges the tasks according to the demand intensity value, and generates a priority queue.

[0009] As a further solution of the present invention, the resource allocation conflict identification module includes: The competition relationship extraction submodule extracts the resource allocation situation in each time period based on the priority queue, including the resource demand of the task and the real-time allocated resource amount, and generates a resource allocation competition relationship table; The conflict index calculation submodule calculates the resource allocation conflict index value according to the resource allocation competition relationship table, sorts the conflict index values ​​in time series, marks the time periods above the conflict threshold, and generates a conflict time period index set; The state identification generating submodule extracts the number identifications of competing tasks in the corresponding time period based on the conflict time period index set, analyzes the resource conflict between tasks, and generates conflict state identification information.

[0010] As a further solution of the present invention, the resource allocation conflict index adopts the formula: ; in, represents the resource allocation conflict index value, Indicates the The resource requirements of each task, Indicates the The amount of resources allocated to each task in real time, Indicates the total number of tasks.

[0011] As a further solution of the present invention, the task migration trajectory generation module includes: The migration path planning submodule extracts the task migration requirements within the conflict time period based on the conflict status identification information, analyzes the network bandwidth and migration cost between nodes, selects the optimal migration path, and generates a migration node sequence; The migration time recording submodule records the migration time range of each task, including the start time and end time, according to the migration node sequence, and generates migration time range information; The trajectory generation submodule calls the migration time range information, integrates the migration path and time range, identifies and labels the continuous segments, and generates task migration trajectory information.

[0012] As a further solution of the present invention, the system also includes a scheduling efficiency evaluation module: The scheduling efficiency evaluation module analyzes the changes in node resource utilization after task migration based on the task migration trajectory information, calculates the migration efficiency factor, marks the time period when the migration efficiency factor is lower than the benchmark value as an inefficient scheduling area, and generates real-time scheduling efficiency identification information; The real-time scheduling efficiency identification information includes an inefficient scheduling time period, a migration efficiency factor, and a resource utilization change.

[0013] As a further solution of the present invention, the scheduling efficiency evaluation module includes: The utilization change analysis submodule extracts resource utilization data of nodes before and after migration based on the task migration trajectory information, identifies utilization changes, and sorts the changes in time series to generate a utilization change set; The efficiency factor calculation submodule identifies the migration efficiency factor value in each time period based on the utilization rate change set, sorts the efficiency factor values ​​in time series, marks the time periods below the benchmark value, and generates an index set of inefficient scheduling time periods; The inefficient area marking submodule extracts the utilization change identifier within the corresponding time period based on the inefficient scheduling time period index set, analyzes the trend and magnitude of the utilization change, and generates real-time scheduling efficiency identification information.

[0014] In another aspect, a cluster computing resource real-time scheduling method is provided. The cluster computing resource real-time scheduling method is performed based on the above cluster computing resource real-time scheduling system and includes the following steps: S1: Obtain the operating status information of the nodes in the computing cluster, extract the processor occupancy rate and memory usage rate of each node, analyze the changing trends of occupancy rate and usage rate, identify the time points when abnormal fluctuations in resource usage occur, and generate a list of resource fluctuation events; S2: Based on the resource fluctuation event list and in combination with the execution characteristic parameters of the tasks, including task type, expected execution time, required processor resources, and memory resources, the resource demand intensity of each task is analyzed, tasks with high demand intensity are screened, and a priority queue is generated; S3: extracting resource contention within a corresponding time period based on the priority queue, identifying whether there is a resource allocation conflict by calculating a resource allocation conflict index, and generating conflict status identification information; S4: Based on the conflict status identification information, identify the task migration path within the conflict time period, record the node sequence and migration time of the task migration, and generate task migration trajectory information; S5: Analyze the resource utilization change of the node after task migration based on the task migration trajectory information, calculate the resource scheduling efficiency after migration, mark the time period when the migration efficiency is lower than the benchmark value, and generate real-time scheduling efficiency identification information.

[0015] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least: By continuously collecting processor and memory utilization rates from compute nodes and analyzing their changing trends, we can accurately pinpoint resource usage fluctuations, enhance awareness of resource status changes, quickly capture sudden load fluctuations, and improve proactive scheduling responses. Correlation analysis between task execution characteristics and resource demand intensity helps screen high-demand tasks and prioritize resource allocation, ensuring the runtime efficiency and overall service quality of critical tasks. Constructing a conflict index based on resource competition relationships identifies conflicts in resource allocation, enhances the adaptive adjustment capabilities of scheduling policies, and mitigates the instability risks associated with resource contention. Task migration paths and time ranges are generated based on conflict periods, ensuring optimal migration path selection and balanced load acceptance on target nodes. By analyzing changes in resource utilization after migration, migration efficiency can be quantified, inefficient periods can be identified, and scheduling policies can be optimized, forming a dynamic corrective closed-loop. The overall scheduling process implements coordinated control across multiple key dimensions, including resource anomaly identification, task screening, conflict avoidance, and efficiency assessment, significantly improving the flexibility, stability, and efficiency of resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 This is a schematic diagram of a real-time scheduling system for cluster computing resources provided by an embodiment of the present invention; Figure 2 Schematic diagram of the system framework of the present invention; Figure 3 This is a flow chart of the resource fluctuation detection module in the present invention; Figure 4 This is a flow chart of the task priority determination module in the present invention; Figure 5 This is a flow chart of the resource allocation conflict identification module in the present invention; Figure 6 This is a flowchart of the task migration trajectory generation module in the present invention; Figure 7 This is a flow chart of the scheduling efficiency evaluation module in the present invention; Figure 8 This is a flow chart of a real-time scheduling method for cluster computing resources provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0019] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0020] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0021] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0022] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0023] The embodiment of the present invention provides a cluster computing resource real-time scheduling system, such as Figure 1-2 The diagram of the cluster computing resource real-time scheduling system shown in FIG. 1 includes: The resource fluctuation detection module obtains the operating status information of the nodes in the computing cluster, analyzes the changing trends of processor occupancy and memory usage, marks the time points when resource usage shifts, and generates a list of resource fluctuation events; The task priority determination module analyzes the resource demand intensity of tasks based on the resource fluctuation event list and the execution characteristic parameters of tasks, selects tasks with high resource demand intensity, and generates a priority queue; The resource allocation conflict identification module extracts the resource competition relationship within the corresponding time period based on the priority queue, determines whether there is a resource allocation conflict state by identifying the resource allocation conflict index, and generates conflict state identification information; The task migration trajectory generation module identifies the task migration path within the conflict time period based on the conflict status identification information, records the node sequence and migration time range of the task migration, and generates task migration trajectory information; The scheduling efficiency evaluation module analyzes the changes in node resource utilization after task migration based on task migration trajectory information, calculates the migration efficiency factor, marks the time period when the migration efficiency factor is lower than the benchmark value as an inefficient scheduling area, and generates real-time scheduling efficiency identification information.

[0024] The resource fluctuation event list includes the resource utilization offset time point, fluctuation amplitude, and utilization change trend. The priority queue includes high-demand intensity task identification, task priority sequence number, and task execution characteristic parameters. The conflict status identification information includes resource allocation conflict time period, conflicting task pair, and conflict index. The task migration trajectory information includes migration path, node sequence, and migration time range. The real-time scheduling efficiency identification information includes inefficient scheduling time period, migration efficiency factor, and resource utilization change.

[0025] Specifically, if Figure 2 、 3 As shown, the resource fluctuation detection module includes: The running status acquisition submodule obtains the running status information of the nodes in the computing cluster, including processor occupancy, memory usage, and network bandwidth data. It binds the running status parameters to time tags, establishes a time series data set, and obtains the node running status time series data. During the operation of a computing cluster, each node continuously reports its own operating status information. For example, a computing cluster contains five nodes, labeled Node-A, Node-B, Node-C, Node-D, and Node-E. At a certain moment, the operating status collection submodule collects processor utilization, memory usage, and network bandwidth data from these five nodes. For Node-A, the collected data is: processor utilization of 25%, memory utilization of 40%, and network bandwidth utilization of 15 Mbps. The operating status collection submodule binds the current timestamp to the collected parameters, for example, 20240730-100000, and then stores the time-tagged operating status parameters in a time series database. Over time, the operating status data of each node is continuously collected and stored, ultimately forming a data set containing the operating status of all nodes over a period of time. For example, the operating status data of all nodes between 20240730-095900 and 20240730-100100 is obtained, thus forming the node operating status time series data.

[0026] The occupancy analysis submodule uses the node operation status time series data to call the processor occupancy and memory usage data points in adjacent time periods, calculates the change rate value between consecutive time points, and performs slope extraction on the change trend curve to generate a set of resource occupancy change intervals; For each node, such as Node-A, extract its processor usage and memory usage data points in two adjacent time periods. Assume that the data of two adjacent time points t1 and t2 are extracted. At t1, the processor usage is 25%, and the memory usage is 40%. At t2, the processor usage is 28%, and the memory usage is 45%. The change rate of the processor usage is calculated as , the rate of change of memory usage is , if the time interval For example, if the processor usage rate is 1 second, the processor usage change rate is 3% / s, and the memory usage change rate is 5% / s. This calculation process is performed for all consecutive time points in the time series data to obtain the instantaneous change rate at each time point. The slope extraction operation is performed on the time series change curve of the processor usage and memory usage of each node. Specifically, the slope of the fitting line is calculated at multiple consecutive time points. If the processor usage at five consecutive time points is 25%, 26%, 27%, 28%, and 29%, respectively, a change trend with an approximate slope of 1% / s is obtained through linear fitting. Time periods with similar change trends (for example, slopes within the range of ±2%) are divided into the same change interval. Each interval records the start time, end time, and resource usage change trend within the time period to generate a resource usage change interval set.

[0027] The fluctuation marking submodule identifies the network bandwidth change rate within the same time period based on the resource utilization change interval set, calculates the deviation between the two types of change rates in each time period, selects the time nodes where the deviation value exceeds the resource fluctuation threshold, and generates a resource fluctuation event list; Traverse each resource usage change interval. For example, from 10:00:05 to 10:00:10, the processor and memory usage of Node-A both show an upward trend. Identify the network bandwidth change rate of Node-A during the same time period. Assuming that the network bandwidth is 15 Mbps at 10:00:05 and 25 Mbps at 10:00:10, the network bandwidth change rate is ,Then, calculate the deviation between the change rate of processor or memory usage (e.g., average or maximum) and the network bandwidth change rate in the same time period. Assuming that the average change rate of processor usage is 1.5% / s and the average change rate of memory usage is 2% / s, the average value is 1.75% / s. Convert this percentage change to a scale similar to bandwidth change (e.g., if the total node bandwidth is 100Mbps, then 1.75% corresponds to a change of 1.75Mbps / s). The calculated deviation value is The resource fluctuation threshold is pre-set, for example, it is set to 0.2. If the calculated deviation value 0.25 exceeds the threshold 0.2, the time node (or time period) is marked as a resource fluctuation event. This process is performed for all resource occupancy change intervals of all nodes, and finally a list of time nodes and related information of all resource fluctuation events exceeding the threshold is generated, thus generating a resource fluctuation event list. The setting of resource fluctuation threshold can refer to the statistical analysis of historical resource fluctuation data, for example, the 90% quantile of the historical deviation value is calculated as the threshold. Assuming that the historical deviation value distribution is as follows: , the 90% quantile is approximately 0.24, and the resource fluctuation threshold can be set to 0.24.

[0028] Specifically, if Figure 2 、 4 As shown, the task priority determination module includes: The demand intensity calculation submodule extracts the task execution characteristics at each time point based on the resource fluctuation event list, including task type, expected execution time, processor resource requirements, and memory resource requirements. It then integrates the data into data groups according to the time series and generates a task feature parameter set. Traverse the time point of each resource fluctuation event in the list. For example, a resource fluctuation event occurred at 10:00:10. Extract the relevant features of each task being executed in the computing cluster before and after this time point. For each task, extract its task type (such as MapReduce task, stream computing task), expected execution time (for example, in seconds, assuming 60 seconds), processor resource requirements (for example, 2 CPU or GPU cores are required) and memory resource requirements (for example, 4GB of memory is required). Integrate the extracted task execution features according to the time series. For each time point where resource fluctuation occurs, form a data group containing the feature parameters of all tasks running at that time. Finally, summarize the task feature parameter data groups at all resource fluctuation time points to generate a task feature parameter set. For example, at 10:00:10, three tasks are running in the cluster: Task A (MapReduce, estimated 60 seconds, 2 CPUs or GPUs, 4 GB of memory), Task B (Stream Computing, estimated 120 seconds, 1 CPU or GPU, 2 GB of memory), and Task C (Data Mining, estimated 300 seconds, 4 CPUs or GPUs, 8 GB of memory). The parameter set will include (10:00:10, TaskA, MapReduce, 60, 2, 4), (10:00:10, TaskB, Stream Computing, 120, 1, 2), and (10:00:10, TaskC, Data Mining, 300, 4, 8).

[0029] The concept of a GPU (graphics processing unit) can be understood through the description of "processor resource demand" and "resource allocation." This system uses a resource fluctuation detection module to collect real-time operating status information from computing nodes, analyze processor (which can be understood as including CPU and GPU) and memory usage, and accurately identify fluctuations in resource usage. Taking into account task characteristics (such as strong demand for processor resources), it prioritizes scheduling for high-intensity computing tasks, including parallel computing or graphics processing tasks that require GPU acceleration. Resource utilization fluctuations are taken into account in task migration, resource conflict identification, and scheduling efficiency evaluation, aiming to optimize the efficient use of computing resources (including GPUs), achieve rational resource allocation, efficient task execution, and improve overall system performance. Therefore, in this solution, GPUs, as a crucial component of computing resources, play a key role in real-time scheduling, resource allocation conflict determination, and task migration path planning. This solution is particularly suitable for high-performance computing scenarios such as big data and AI training.

[0030] The priority screening submodule calculates the demand intensity value of each task based on the task characteristic parameter set, sorts the demand intensity values ​​in time series, marks tasks with a standard deviation higher than the average demand intensity, and generates a demand intensity task set; For each task, the demand intensity value of the task is calculated based on its characteristic parameters such as task type, expected execution time, processor resource requirement, and memory resource requirement. The demand intensity value can be calculated by weighted summation. For example, set the processor resource requirement weight to 0.4, the memory resource requirement weight to 0.3, the expected execution time weight to 0.2, and the task type weight to 0.1 (the task type can be quantified, such as MapReduce is 1, stream computing is 2, and data mining is 3). For task A (2 CPUs or GPUs, 4 GB of memory, 60 seconds, MapReduce), assuming its standardized processor requirement is 0.5, memory requirement is 0.6, the expected execution time is standardized to 0.2, and the task type is quantified to 1, then the demand intensity value = Calculate the demand intensity values ​​for all tasks in the same way. Then, sort the demand intensity values ​​of all tasks in time series, calculate the average demand intensity value of all tasks over a period of time, and calculate its standard deviation. Tasks with a demand intensity value higher than the average demand intensity plus one standard deviation are labeled as high-demand intensity tasks. Finally, a set of task identification indexes and demand intensity values ​​containing high-demand intensity tasks is generated, forming a demand intensity task set. For example, if the demand intensity values ​​of all tasks over a period of time are as follows: 0.3, 0.45, 0.52, 0.28, 0.6, and 0.35, with an average of 0.42 and a standard deviation of 0.15, then the only tasks with a demand intensity higher than 0.42 + 0.15 = 0.57 are tasks with a demand intensity of 0.6.

[0031] The queue generation submodule extracts the task identification index and demand intensity value based on the demand intensity task set, arranges them according to the demand intensity value, and generates a priority queue; Traverse each high-demand intensity task in the set, extract the unique identification index of each task (such as TaskA-ID) and its calculated demand intensity value (such as 0.6), and then sort the tasks in descending order according to their demand intensity values ​​to generate a priority queue. Each element in the queue contains the task identification index and the corresponding demand intensity value. The higher the demand intensity value, the higher the task is in the queue. In this way, a queue is formed according to the demand intensity of the tasks. For example, the demand intensity task set contains (TaskC-ID, 0.75), (TaskA-ID, 0.6), (TaskE-ID, 0.82), then the generated priority queue is (TaskE-ID, 0.82), (Taskc-ID, 0.75), (TaskA-ID, 0.6).

[0032] Specifically, if Figure 2 、 5 As shown, the resource allocation conflict identification module includes: The competition relationship extraction submodule extracts the resource allocation situation in each time period based on the priority queue, including the task's demand for resources and the amount of resources allocated in real time, and generates a resource allocation competition relationship table; For each task in the priority queue, such as the first task in the queue, Task E-ID, its resource allocation status in the current time period is extracted. This includes Task E-ID's demand for various resources (for example, 4 CPU or GPU cores, 8GB of memory) and the amount of resources allocated to the task by the cluster in real time (for example, only 3 CPU or GPU cores and 6GB of memory are currently allocated). This process is performed on all tasks in the priority queue in the same time period, forming a table that records each task's resource demand and actual allocation status, namely the resource allocation competition relationship table. This table contains fields such as task ID, required number of processor cores, allocated number of processor cores, required memory capacity, and allocated memory capacity.

[0033] The conflict index calculation submodule calculates the resource allocation conflict index value based on the resource allocation competition relationship table, sorts the conflict index values ​​in time series, marks the time periods above the conflict threshold, and generates a conflict time period index set; Resource allocation conflict index, using the formula: ; in, represents the resource allocation conflict index value, Indicates the The resource requirements of each task, Indicates the The amount of resources allocated to each task in real time, Indicates the total number of tasks; The resource allocation conflict index is a key indicator for measuring whether there is resource competition between different tasks or processes during the computing resource allocation process. By analyzing the computing resource demands (such as processors and memory) of different tasks within the same time period, the severity of the conflict is calculated. A higher index value indicates that more tasks are competing for limited resources at the same time, resulting in performance degradation or task delays. The resource allocation conflict index can effectively identify periods of unbalanced resource usage, thereby adjusting task scheduling strategies and optimizing resource utilization. Traverse each record in the table, that is, each task, and extract the resource requirements of the task and the amount of resources allocated in real time , calculate the difference between the demand and allocation for processor resources and memory resources respectively, assuming that the total number of tasks For TaskE-ID, the processor requirement is 5. For 4, the distribution amount 3, memory requirement 8, the distribution amount If is 6, the processor resource conflict of this task is , memory resource conflict is , sum up the processor resource conflicts and memory resource conflicts of all tasks respectively, and get the total processor resource conflict Conflict with total memory resources , resource allocation conflict index The following formula is used for calculation: represents the resource allocation conflict index value, Indicates the The processor demand of each task, Indicates the The amount of processors to which each task is assigned in real time, Indicates the The memory requirement of each task, Indicates the The amount of memory allocated to each task in real time, Indicates the total number of tasks. Assuming there are currently 3 tasks, their resource requirements and allocation are as follows: Task 1 (CPU or GPU requires 4, allocate 3, memory requires 8, allocate 6); Task 2 (CPU or GPU requires 2, allocate 2, memory requires 4, allocate 3); Task 3 (CPU or GPU requires 6, allocates 4, memory requires 12, allocates 8); The total number of processors required is 4+2+6=12, the total number of allocated processors is 3+2+4=9, the total number of memory required is 8+4+12=24, the total number of allocated memory is 6+3+8=17, and the resource allocation conflict index is: ,After calculating the conflict index values, the conflict index values ​​are sorted in time series, and a ,conflict threshold is pre-set, for example, 0.3. Time periods above the threshold are marked as time periods with ,more serious resource allocation conflicts, and a conflict time period index set ,is generated.

[0034] The state identification generation submodule extracts the number of competing tasks in the corresponding time period based on the conflict time period index set, analyzes the resource conflicts between tasks, and generates conflict state identification information; Traverse each conflict time period in the index set, such as 10:00:10-10:00:15, and extract the number of tasks that have resource competition in this time period. Specifically, count the number of tasks whose demand for resources is greater than the actual allocated resource amount in this time period. For example, during the period of 10:00:10-10:00:15, there are 3 tasks whose demand is greater than the allocated amount, so the number of competing tasks is identified as 3. Analyze the specific conflicts between tasks for processor, memory and other resources, such as which tasks compete for the same resources at the same time and the degree of competition. Based on the number of competing tasks and the type and quantity of competing resources, generate identification information describing the current resource conflict status, such as "high CPU or GPU memory contention, 3 tasks conflicting".

[0035] Specifically, if Figure 2 、 6 As shown in Figure 2, the task migration trajectory generation module includes: The migration path planning submodule extracts the task migration requirements within the conflict time period based on the conflict status identification information, analyzes the network bandwidth and migration cost between nodes, selects the optimal migration path, and generates a migration node sequence; Extract the time periods where resource conflicts occur and the list of tasks requiring task migration. For example, between 10:00:10 and 10:00:15, there is a resource conflict between Task A and Task C. Analyze the network bandwidth between nodes in the computing cluster and the migration cost of migrating tasks from one node to another (such as the time required for migration and the network resources occupied). Select the optimal migration path based on the network bandwidth and migration cost. The optimal path can be selected based on a comprehensive consideration of factors such as minimizing migration time and minimizing the impact on tasks. For example, if Task A can migrate from Node-1 to Node-3 or Node-4, and the network bandwidth from Node-1 to Node-3 is high and the migration cost is low, then select Node-3 as the target node. Generate a sequence for each task requiring migration, including its original node and target node, to generate a migration node sequence, such as (TaskA, Node-1 → Node-3) and (TaskC, Node-2 → Node-5).

[0036] The migration time recording submodule records the migration time range of each task, including the start time and end time, based on the migration node sequence, and generates migration time range information; According to the migration path of each task in the sequence, record the migration start time and migration end time of each task. For example, if Task A starts migrating from Node-1 to Node-3 at 10:00:20 and completes at 10:00:25, its migration time range is 10:00:20-10:00:25. This process records the start and end time of all migration tasks and generates migration time range information, such as (TaskA, [10:00:20, 10:00:25), (Taskc, [10:00:22, 10:00:28]).

[0037] The trajectory generation submodule calls the migration time range information, integrates the migration path and time range, identifies and marks the continuous segments, and generates task migration trajectory information; Call information, integrate the migration path of each task (for example, Node-1->Node-3) and the corresponding migration time range (for example, 10:00:20-10:00:25), identify migration segments that are continuous in time, for example, if multiple tasks migrate from some nodes to other nodes in a similar time period, then identify the continuous migration behavior, and finally generate task migration trajectory information containing the complete migration path and time information of each task, for example, TaskA migrates from Node-1 to Node-3 at 10:00:20-10:00:25, TaskC at 10:00:22 If TaskA migrates from Node-2 to Node-5 at 10:00:28, the trajectory information is represented as (TaskA, Node-1→Node-3, 10:00:20−10:00:25), (TaskC, Node-2→Node-5, 10:00:22−10:00:28). If TaskB subsequently migrates from Node-3 to Node-4 between 10:00:28 and 10:00:32, it can be identified that there is a continuous segment of task migration in and out between 10:00:20 and 10:00:32 on Node-3 and marked.

[0038] Specifically, if Figure 2 、 7 As shown, the scheduling efficiency evaluation module includes: The utilization change analysis submodule extracts resource utilization data of nodes before and after migration based on task migration trajectory information, identifies utilization changes, and sorts the changes in time series to generate a utilization change set. For each task that is migrated, extract the resource utilization data of the source node before migration and the target node after migration. For example, before Task A is migrated from Node-1 to Node-3, the CPU or GPU utilization of Node-1 is 70%, and the memory utilization is 80%. The CPU or GPU utilization of Node-3 is 50%, and the memory utilization is 60%. After the migration is completed, suppose that after a period of time, the CPU or GPU utilization of Node-1 becomes 55%, and the memory utilization is 65%, and the CPU or GPU utilization of Node-3 becomes 65%. %, and memory utilization is 75%. Calculate the change in resource utilization for each node before and after migration. For example, the CPU or GPU utilization of Node-1 changes to 55% − 70% = − 15%, and the memory utilization changes to 65% − 80% = − 15%. The CPU or GPU utilization of Node-3 changes to 65% − 50% = + 15%, and the memory utilization changes to 75% − 60% = + 15%. Sort the utilization changes by time series to generate a time series data set containing the changes in node resource utilization caused by all migration events, and then generate a utilization change set.

[0039] The efficiency factor calculation submodule identifies the migration efficiency factor value within each time period based on the utilization change set, sorts the efficiency factor values ​​in time series, marks the time periods that are lower than the benchmark value, and generates an index set of inefficient scheduling time periods; Traverse each time period (or each migration event) in the set and identify the efficiency factor value of the task migration that occurred in the time period. The calculation of the efficiency factor can be quantitatively evaluated based on factors such as the balance of resource utilization after migration and the improvement of task execution efficiency. For example, the ratio of the standard deviation of the resource utilization of each node after migration to the standard deviation of the resource utilization of each node before migration can be calculated as the efficiency factor. If the ratio is less than 1, it is considered that the migration has improved the balance of resource utilization and the efficiency is high. Otherwise, it is low. Assume that the CPU or GPU utilization of Node-1 and Node-3 before migration is 70% and 50% respectively, and the standard deviation is , after migration, they are 55% and 65% respectively, with an average of 60% and a standard deviation of , then the efficiency factor is , sort the calculated efficiency factor values ​​in time series, pre-set a benchmark efficiency factor value, such as 0.8, and mark the time periods corresponding to efficiency factors below this benchmark value as inefficient scheduling time periods, generating an index set of inefficient scheduling time periods. The benchmark efficiency factor value can be set by statistically analyzing historical scheduling efficiency data. For example, the 10% quantile of the historical efficiency factor can be calculated as the benchmark value. Assuming the historical efficiency factor distribution is as follows: 0.95, 0.9, 0.85, 0.78, 0.7, 0.65, the 10% quantile is approximately 0.68, so the benchmark efficiency factor can be set to 0.68.

[0040] The inefficient area marking submodule extracts the utilization change identifier within the corresponding time period based on the inefficient scheduling time period index set, analyzes the trend and magnitude of the utilization change, and generates real-time scheduling efficiency identification information; Traverse each inefficient scheduling time period in the index set and extract the resource utilization change identifier that occurred during this time period. For example, during a certain inefficient scheduling time period, the CPU or GPU utilization of Node-1 decreased significantly (-20%), while the CPU or GPU utilization of Node-3 only increased slightly (+5%). This indicates that task migration did not effectively balance the resource load. Analyze the trend and magnitude of this utilization change to determine whether there are problems such as excessive migration and improper target node selection. Finally, generate scheduling efficiency identification information, such as "During the period of 10:00:20-10:00:25, the CPU or GPU utilization of Node-1 decreased significantly, while the CPU or GPU utilization of Node-3 did not increase significantly, indicating low scheduling efficiency."

[0041] See also Figure 8 The cluster computing resource real-time scheduling method is based on the above cluster computing resource real-time scheduling system and includes the following steps: S1: Obtain the operating status information of the nodes in the computing cluster, extract the processor occupancy rate and memory usage rate of each node, analyze the changing trends of occupancy rate and usage rate, identify the time points when abnormal fluctuations in resource usage occur, and generate a list of resource fluctuation events; S2: Based on the resource fluctuation event list and combined with the execution characteristic parameters of the task, including task type, expected execution time, required processor resources and memory resources, analyze the resource demand intensity of each task, screen the tasks with the highest demand intensity, and generate a priority queue; S3: Extract resource contention within the corresponding time period based on the priority queue, calculate the resource allocation conflict index to identify whether there is a resource allocation conflict, and generate conflict status identification information; S4: Based on the conflict status identification information, identify the task migration path within the conflict time period, record the node sequence and migration time of the task migration, and generate task migration trajectory information; S5: Based on the task migration trajectory information, analyze the changes in node resource utilization after task migration, calculate the resource scheduling efficiency after migration, mark the time period when the migration efficiency is lower than the benchmark value, and generate real-time scheduling efficiency identification information.

[0042] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A real-time scheduling system for cluster computing resources, characterized in that: The system comprises: The resource fluctuation detection module obtains the operating status information of the nodes in the computing cluster, analyzes the changing trends of processor occupancy and memory usage, marks the time points when resource usage shifts, and generates a list of resource fluctuation events; The task priority determination module analyzes the resource demand intensity of the task based on the resource fluctuation event list and the execution characteristic parameters of the task, selects tasks with high demand intensity, and generates a priority queue; The resource allocation conflict identification module extracts the resource competition relationship in the corresponding time period according to the priority queue, determines whether there is a resource allocation conflict state by identifying the resource allocation conflict index, and generates conflict state identification information; The task migration trajectory generation module identifies the task migration path within the conflict time period based on the conflict status identification information, records the node sequence and migration time range of the task migration, and generates task migration trajectory information.

2. The cluster computing resource real-time scheduling system according to claim 1, characterized in that: The resource fluctuation event list includes the resource utilization offset time point, fluctuation amplitude, and utilization change trend; the priority queue includes high-demand intensity task identification, task priority sequence number, and task execution characteristic parameters; the conflict status identification information includes resource allocation conflict time period, conflicting task pair, and conflict index; the task migration trajectory information includes migration path, node sequence, and migration time range.

3. The real-time cluster computing resource scheduling system according to claim 1, characterized in that: The resource fluctuation detection module includes: The running status acquisition submodule obtains the running status information of the nodes in the computing cluster, including processor occupancy, memory usage, and network bandwidth data. It binds the running status parameters to time tags, establishes a time series data set, and obtains the node running status time series data. The occupancy analysis submodule calls the processor occupancy and memory usage data points in adjacent time periods based on the node operation status time series data, calculates the change rate value between consecutive time points, and performs slope extraction on the change trend curve to generate a resource occupancy change interval set; The fluctuation marking submodule identifies the network bandwidth change rate within the same time period based on the resource occupancy rate change interval set, calculates the deviation value of the two types of change rates in each time period, filters the time nodes where the deviation value exceeds the resource fluctuation threshold, and generates a resource fluctuation event list.

4. The cluster computing resource real-time scheduling system according to claim 3, characterized in that: The task priority determination module includes: The demand intensity calculation submodule extracts the task execution characteristics at each time point based on the resource fluctuation event list, including task type, expected execution time, processor resource requirements, and memory resource requirements, and integrates them into data groups according to time series to generate a task characteristic parameter set; The priority screening submodule calculates the demand intensity value of each task based on the task characteristic parameter set, sorts the demand intensity values ​​in time series, marks tasks with a standard deviation higher than the average demand intensity, and generates a demand intensity task set; The queue generation submodule extracts the task identification index and the demand intensity value based on the demand intensity task set, arranges the tasks according to the demand intensity value, and generates a priority queue.

5. The cluster computing resource real-time scheduling system according to claim 4, characterized in that: The resource allocation conflict identification module includes: The competition relationship extraction submodule extracts the resource allocation situation in each time period based on the priority queue, including the resource demand of the task and the real-time allocated resource amount, and generates a resource allocation competition relationship table; The conflict index calculation submodule calculates the resource allocation conflict index value according to the resource allocation competition relationship table, sorts the conflict index values ​​in time series, marks the time periods above the conflict threshold, and generates a conflict time period index set; The state identification generating submodule extracts the number identifications of competing tasks in the corresponding time period based on the conflict time period index set, analyzes the resource conflict between tasks, and generates conflict state identification information.

6. The cluster computing resource real-time scheduling system according to claim 5, characterized in that: The resource allocation conflict index adopts the formula: ； in, represents the resource allocation conflict index value, Indicates the The resource requirements of each task, Indicates the The amount of resources allocated to each task in real time, Indicates the total number of tasks.

7. The cluster computing resource real-time scheduling system according to claim 5, characterized in that: The task migration trajectory generation module includes: The migration path planning submodule extracts the task migration requirements within the conflict time period based on the conflict status identification information, analyzes the network bandwidth and migration cost between nodes, selects the optimal migration path, and generates a migration node sequence; The migration time recording submodule records the migration time range of each task, including the start time and end time, according to the migration node sequence, and generates migration time range information; The trajectory generation submodule calls the migration time range information, integrates the migration path and time range, identifies and labels the continuous segments, and generates task migration trajectory information.

8. The cluster computing resource real-time scheduling system according to claim 1, characterized in that: The system also includes a scheduling efficiency evaluation module: The scheduling efficiency evaluation module analyzes the changes in node resource utilization after task migration based on the task migration trajectory information, calculates the migration efficiency factor, marks the time period when the migration efficiency factor is lower than the benchmark value as an inefficient scheduling area, and generates real-time scheduling efficiency identification information; The real-time scheduling efficiency identification information includes an inefficient scheduling time period, a migration efficiency factor, and a resource utilization change.

9. The cluster computing resource real-time scheduling system according to claim 8, characterized in that: The scheduling efficiency evaluation module includes: The utilization change analysis submodule extracts resource utilization data of nodes before and after migration based on the task migration trajectory information, identifies utilization changes, and sorts the changes in time series to generate a utilization change set; The efficiency factor calculation submodule identifies the migration efficiency factor value in each time period based on the utilization rate change set, sorts the efficiency factor values ​​in time series, marks the time periods below the benchmark value, and generates an index set of inefficient scheduling time periods; The inefficient area marking submodule extracts the utilization change identifier within the corresponding time period based on the inefficient scheduling time period index set, analyzes the trend and magnitude of the utilization change, and generates real-time scheduling efficiency identification information.

10. A real-time scheduling method for cluster computing resources, characterized in that: The method is used to implement the cluster computing resource real-time scheduling system according to any one of claims 1 to 9, comprising the following steps: S1: Obtain the operating status information of the nodes in the computing cluster, extract the processor occupancy rate and memory usage rate of each node, analyze the changing trends of occupancy rate and usage rate, identify the time points when abnormal fluctuations in resource usage occur, and generate a list of resource fluctuation events; S2: Based on the resource fluctuation event list and in combination with the execution characteristic parameters of the tasks, including task type, expected execution time, required processor resources, and memory resources, the resource demand intensity of each task is analyzed, tasks with high demand intensity are screened, and a priority queue is generated; S3: extracting resource contention within a corresponding time period based on the priority queue, identifying whether there is a resource allocation conflict by calculating a resource allocation conflict index, and generating conflict status identification information; S4: Based on the conflict status identification information, identify the task migration path within the conflict time period, record the node sequence and migration time of the task migration, and generate task migration trajectory information; S5: Analyze the resource utilization change of the node after task migration based on the task migration trajectory information, calculate the resource scheduling efficiency after migration, mark the time period when the migration efficiency is lower than the benchmark value, and generate real-time scheduling efficiency identification information.

Images