Server cluster scheduling method based on dynamic load balancing

Abstract

The invention belongs to the technical field of server cluster scheduling, and particularly relates to a dynamic load balancing-based server cluster scheduling method, which comprises the following steps of: acquiring load data of each server in a server cluster in real time; performing quantitative evaluation on the acquired load data through a preset load evaluation model to obtain a real-time load value and a load stability score of each server; receiving an external task to be processed, and analyzing resource demand parameters and task type characteristics of the task; determining a target server of the task based on the server state level, the load stability score, the task resource demand parameter and the task type feature; and updating the load evaluation model and the scheduling strategy in real time based on the historical scheduling data, the task operation feedback data and the industry scene characteristic parameters. According to the method, through multi-dimensional load evaluation, accurate matching of tasks and servers and dynamic strategy optimization, the resource utilization rate and task processing efficiency of the server cluster are effectively improved, and the requirements of different industry scenes are met.

Description

Technical Field

[0001] The present invention belongs to the technical field of server cluster scheduling, and in particular relates to a server cluster scheduling method based on dynamic load balancing. Background Art

[0002] Amid the rapid development of internet technology, server clusters have become the core infrastructure supporting large-scale concurrent task processing, widely used in scenarios such as e-commerce platforms, financial transactions, and medical data processing. With the diversification of user needs and the increasing complexity of tasks, the types of tasks being processed in clusters have shown significant diversity, ranging from compute-intensive data analysis to I / O-intensive file storage. Furthermore, task resource demands fluctuate dynamically, such as the high concurrency of orders during e-commerce promotions and the peak transaction requests on financial trading days. This places extremely high demands on the load balancing and scheduling capabilities of server clusters.

[0003] Traditional server cluster scheduling methods have the following technical defects: The load assessment dimension is single and the status judgment accuracy is insufficient: Existing load assessment technologies often rely on dynamic load parameters such as real-time CPU usage and memory utilization. These only reflect the current load intensity of the server and fail to quantify load fluctuations. For example, large fluctuations in load over a short period of time can cause task interruptions. For example, some servers may be deemed idle due to a momentary load reduction, but in reality, they experience high-frequency load fluctuations. After a task is assigned, they can quickly become overloaded, resulting in reduced task processing efficiency.

[0004] Task scheduling has poor adaptability, and resource utilization and task efficiency are unbalanced: Existing scheduling strategies often rely on a simple rule of prioritizing idle servers, failing to consider the compatibility between task types and server performance characteristics. For example, they assign compute-intensive tasks to servers with ample memory redundancy but weak CPU performance. Alternatively, they fail to distinguish between task priority and the need for server stability due to runtime. For example, assigning long tasks to servers with volatile loads can easily lead to task interruptions due to mid-process load migration. Furthermore, when faced with high-priority tasks, traditional methods often directly occupy the remaining resources of healthy servers without establishing a dynamic migration mechanism for overloaded servers. This can lead to rapid load saturation on healthy servers, triggering the risk of cascading overloads. Static strategies make it difficult to adapt to scenario-based needs and dynamic changes: There are significant differences in the task characteristics of different industry scenarios. For example, some scenarios are sensitive to task response delays, and e-commerce scenarios have high requirements for high concurrency carrying capacity. However, existing scheduling strategies mostly use general rules and cannot be dynamically adjusted according to scenario characteristics. For example, resources are not prioritized for short tasks of e-commerce promotions, or emergency resource pools are not reserved for financial transaction tasks.

[0005] Imperfect load migration and emergency dispatch mechanisms: Existing task migration strategies for server overloads are often based on a simple logic of "randomly selecting low-priority tasks," failing to consider inter-task dependencies. For example, splitting and migrating related tasks can lead to data consistency issues and migration costs. For example, the migration of large file transfers can be time-consuming, impacting the immediate allocation of high-priority tasks. Furthermore, in extreme scenarios, such as when the overall cluster load exceeds the upper limit, the lack of a coordinated scheduling mechanism with edge nodes makes it difficult to alleviate pressure on the core cluster by expanding the resource pool, leading to task backlogs and service interruptions. Therefore, how to build a scheduling method that can accurately evaluate server load status, dynamically adapt to task types and scenario requirements, and have self-optimization capabilities has become a key technical issue in improving server cluster resource utilization and task processing efficiency. Summary of the Invention

[0006] The purpose of the present invention is to provide a server cluster scheduling method based on dynamic load balancing, which is used to accurately evaluate the server load status, dynamically adapt task types and scenario requirements, and realize self-optimization scheduling to improve server cluster resource utilization and task processing efficiency.

[0007] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows: A server cluster scheduling method based on dynamic load balancing includes the following steps: S1: Collect load data of each server in the server cluster in real time. The load data includes static load parameters, dynamic load parameters, and load fluctuation characteristic parameters. S2: A preset load assessment model quantitatively evaluates load data, obtaining the real-time load value and load stability score for each server. Based on the real-time load value, the server is classified into three status levels: idle, normal, and overloaded. The load stability score is used to indicate the severity of server load fluctuations within a preset time period. S3: Receive external tasks to be processed and analyze the resource requirement parameters and task type characteristics of the tasks to be processed; S4: Determine the target server for the task through a preset scheduling strategy based on the server's status level, load stability score, task resource requirement parameters, and task type characteristics; S5: Assign the task to the corresponding target server and monitor the load changes and task progress matching of the target server in real time during the task operation; if the target server load value exceeds the preset overload threshold or the task progress matching is lower than the preset threshold, perform load migration; S6: Based on historical scheduling data, task operation feedback data and industry scenario characteristic parameters, the load assessment model and scheduling strategy are updated in real time.

[0008] Preferably, the static load parameters are the inherent attribute parameters of the specified hardware of the server, the dynamic load parameters are the performance parameters that change in real time when the server is running, and the load fluctuation characteristic parameters are the fluctuation amplitude, frequency and trend slope of the dynamic load within a preset time period; The task progress matching degree is the deviation rate between the actual task progress and the preset progress; The resource requirement parameters include the CPU usage, memory usage, network bandwidth, and task priority required by the task; the task type characteristics include whether the task is compute-intensive, IO-intensive, or mixed, and the expected runtime of the task; The industry scenario characteristic parameters include the distribution pattern of task types within the scenario, peak period distribution and common characteristics of resource demand.

[0009] Preferably, the scheduling strategy includes: Prioritize assigning tasks to servers that are idle and have a load stability score ≥ the preset threshold; For computing-intensive tasks, servers with high CPU performance redundancy are prioritized, and for IO-intensive tasks, servers with high disk IO throughput are prioritized. If the task is of high priority and the normal state server has insufficient resources, the task is assigned after triggering the load migration of the overloaded state server.

[0010] Preferably, the load fluctuation characteristic parameters are calculated by a sliding window algorithm: a sliding window is taken in units of 5 minutes, the standard deviation of the dynamic load parameters in the statistical window is taken as the fluctuation amplitude, the number of peak occurrences is taken as the fluctuation frequency, and the linear fitting slope is taken as the fluctuation trend; Among them, when the fluctuation trend has a positive slope, it indicates that the load is on an upward trend, and when it has a negative slope, it indicates that the load is on a downward trend.

[0011] Preferably, the specific process of quantitatively evaluating the load data using the load evaluation model preset in step S2 to obtain the real-time load value and load stability score of each server is as follows: S21: Preprocessing the load data of each server; S22: Preset basic weights based on server type: Assign fixed basic weights to static load parameters including the number of CPU cores and total memory capacity; Assign dynamic basic weights to dynamic load parameters including real-time CPU usage and memory usage; The load fluctuation characteristic parameters including fluctuation amplitude, frequency and trend slope are respectively assigned basic weights for the load stability score; S23: Calculate the real-time load value to quantify the current load intensity of the server: Static load parameter quantification: Based on the maximum load capacity of the server hardware, the static parameters are converted into a quantitative value of hardware redundancy: Static parameter quantification value = (maximum hardware specification - currently allocated specification) / maximum hardware specification; Dynamic load parameter quantification: Focusing on the server's real-time performance utilization, the weights are dynamically adjusted based on load fluctuation characteristics. Basic quantization: dynamic parameter quantization value = 1-real-time usage rate; Weight correction: If the load fluctuation range is greater than the preset threshold, the weight of the dynamic parameter is reduced, while the weight of the parameter with higher stability is increased; Calculate the real-time load value by weighted sum: Real-time load value = (static parameter quantization value × static weight) + (dynamic parameter quantization value × dynamic corrected weight); S24: Calculate the load stability score to quantify the risk of server load fluctuation: Quantification of load fluctuation characteristic parameters: The volatility data calculated based on a 5-minute sliding window is converted into quantitative values: Fluctuation amplitude: based on the standard deviation of the dynamic load parameter, standardized to a score of 0-10; Fluctuation frequency: counts the number of times exceeding the preset threshold within 5 minutes, normalized to a score of 0-10; Trend slope: Convert the load change trend slope into 0-10 points; The stability score is calculated according to the preset weights: Load stability score = (fluctuation amplitude score × 40%) + (fluctuation frequency score × 30%) + (trend slope score × 30%).

[0012] Preferably, the specific process of step S3 is as follows: S31: Perform preliminary verification on the received tasks to filter out invalid or abnormal tasks, including legality verification and integrity verification; S32: Based on the resource description information carried by the task, that is, the parameters specified when the task is submitted or the preset task template, extract the four core resource requirements, including CPU requirements, memory requirements, network bandwidth requirements, and storage I / O requirements; S33: Task priority analysis and dynamic adjustment: Basic priority extraction: read the preset priority levels P0-P3 from the task parameters; Dynamic priority adjustment: Adjustment based on task submission time and timeout requirements; S34: Initial judgment based on task behavior type: Preliminary classification based on initial task information: Computationally intensive tasks: If the task involves a large amount of mathematical operations, no frequent disk reads and writes, and little network interaction, it is initially judged to be computationally intensive and marked with a CPU-sensitive label; I / O-intensive tasks: If a task involves a large amount of file reading and writing, database queries, or network data transmission, and the CPU usage is lower than the preset value, it is determined to be I / O-intensive and marked with a disk I / O / network sensitive label; Hybrid tasks: If they meet the characteristics of both computation-intensive and computation-intensive tasks, they will be marked with CPU+IO dual-sensitive labels; Secondary verification of features based on trial runs: For the first appearance of an unknown type of task, a micro-sample trial run mechanism is initiated: minimal resources are allocated to allow the task to run the first 10% of the process, and the resource consumption curve of this stage is collected; Task duration and correlation analysis: Runtime: Analyze the estimated execution time of the task or estimate it based on historical data, and categorize the duration into short tasks, medium tasks, and long tasks; Relevance: Check whether the task is part of a task chain. If there is a dependency, record the associated task ID. During subsequent scheduling, ensure that the associated tasks are assigned to the same subnet server with a network latency of ≤10ms.

[0013] Preferably, the specific process of matching tasks with target servers in a hierarchical screening manner according to the preset scheduling policy in step S4 is as follows: S41: Filter idle servers, servers with load values ​​≤ 30%: If there is at least one server after screening, the server ranked first is selected as the target server in descending order of the comprehensive score of resource matching degree + type adaptability; if there is no idle server that meets the conditions, step S42 is executed; S42: Filter servers in normal status, servers with a load value of 30% < ≤ 70%: If there are servers that meet the conditions after screening, the target server is determined in the order of priority adaptability > resource buffer space > stability score; if the server resources are insufficient in the normal state, such as there is no matching server for the high-priority task, step S43 is executed; S43: Overloaded server scheduling, only for high-priority tasks: If the overloaded server meets the task requirements after migration, it will be determined as the target server; if the migration fails, such as there is no migration target server, or the overloaded server load value is ≥85% and cannot be migrated, the cluster expansion warning will be triggered and emergency scheduling will be initiated.

[0014] Preferably, the specific process of step S41 is as follows: S411: First perform basic resource matching: Filter candidate servers that meet the task resource requirement threshold from idle servers: Matching degree calculation: Calculate the matching degree of a single resource by "(server available resources - task requirements) / task requirements"; Eliminate invalid candidates: If the matching degree of a certain resource on the server is less than 0, it will be directly excluded; S412: Perform task type adaptation: From the servers that have passed the basic matching, further filter by task type characteristics: Computation-intensive tasks: Prioritize servers ranked in the top 30% in terms of CPU redundancy, and exclude servers with CPU fluctuations greater than 20% in the past hour. IO-intensive tasks: Prioritize IOPS availability ≥ 60%; Hybrid tasks: Calculate the combined score of "CPU matching degree × 0.4 + I / O matching degree × 0.4 + stability score × 0.2" and select the top 50% of servers. S413: Adapting stability and task duration: Combined with the load stability score and task duration to optimize the selection: Long missions longer than 2 hours: only select servers with a stability score of 7 or higher. Short tasks shorter than 10 minutes: Select from servers with a stability score of 5 or higher, prioritizing servers with the highest resource match. There is no need to wait for highly stable servers to release resources.

[0015] Preferably, the specific process of step S42 is as follows: S421: Strict verification of resource redundancy: In normal state, servers need to reserve more buffer resources to avoid being overloaded after scheduling. Therefore, the resource matching standards are improved: Core rules: Server available resources must be ≥ 1.2 times the task requirements; Special check: If the task is high priority, P0 / P1, additionally check the server load trend slope. If the slope is greater than 0.3, it is excluded even if the resources match. S422: Task priority and server load balancing adaptation: High-priority tasks: Skip the comprehensive score sorting and directly select the server with a resource matching degree ≥ 0.8 and the lowest current load value; Normal-priority tasks: Calculate the server's current load value + the expected increase in load value of the task; Type feature secondary adaptation: S423: Differentiated use of load stability scores for servers in normal state: If the task is I / O-intensive and requires frequent disk reads and writes, select a server with a stability score of 8 or higher. If the task is computationally intensive and requires continuous CPU usage, a server with a trend slope score of ≥ 7 in the stability score is selected. Preferably, the specific process of step S43 is as follows: S431: Overloaded Server Screening and Load Migration Preparation: Filtering criteria: Only servers that are overloaded but not at their limit are selected, with a load value of 70%-85%, no automatic protection mechanism triggered, and server hardware specifications ≥ 1.5 times the task requirements; Migration target determination: Select nodes that can undertake the migration task from idle / normal servers. The nodes must meet the migration task resource requirements and the current load must be ≤ 70%. Prioritize servers in the same subnet as the overloaded servers. S432: Load Migration Execution and Task Allocation: Migration task selection: Based on the principles of low priority and low migration cost; Task assignment timing: After the migration is completed, the current high-priority task will be immediately assigned to the server and marked as temporarily protected. No new tasks will be assigned within the specified time.

[0016] The beneficial effects of the present invention include: The server cluster scheduling method based on dynamic load balancing provided by the present invention collects the load data of each server in the server cluster in real time; quantitatively evaluates the collected load data through a preset load assessment model to obtain the real-time load value and load stability score of each server; receives external tasks to be processed, analyzes the resource requirement parameters and task type characteristics of the tasks; determines the target server of the task based on the server status level, load stability score, task resource requirement parameters and task type characteristics; and updates the load assessment model and scheduling strategy in real time based on historical scheduling data, task operation feedback data and industry scenario characteristic parameters. Through multi-dimensional load assessment, precise matching of tasks and servers and dynamic strategy optimization, the present invention effectively improves the resource utilization and task processing efficiency of the server cluster and adapts to the needs of different industry scenarios.

[0017] First, by simultaneously collecting static load parameters, dynamic load parameters, and load fluctuation characteristic parameters, and combining them with a load assessment model to calculate real-time load values ​​and load stability scores, this breaks through the traditional assessment model that relies solely on real-time load values. On the one hand, the introduction of static parameters accurately reflects the redundancy of server hardware, avoiding the mismatch between task requirements and hardware. On the other hand, the load stability score can identify servers with severe load fluctuations, reducing the risk of misidentifying such servers as idle and assigning tasks, thereby reducing the risk of server overload during task execution.

[0018] Secondly, by analyzing the resource requirements and task type characteristics of each task, and combining them with scheduling strategies, we achieve targeted allocation: compute-intensive tasks are prioritized for servers with high CPU redundancy, I / O-intensive tasks are prioritized for servers with optimal disk I / O, and long tasks are prioritized for servers with high stability. This targeted matching of task characteristics and server capabilities improves task processing efficiency. Furthermore, a mechanism in which high-priority tasks trigger the migration of load from overloaded servers ensures that urgent tasks can be executed quickly even when resources are limited.

[0019] Third, during the task execution phase, real-time monitoring of target server load and task progress matches can trigger load migration in a timely manner, prioritizing low-priority, low-cost tasks to minimize the impact of migration on normal tasks. Real-time updates to load assessment models and scheduling strategies based on historical data, task feedback, and industry scenario parameters ensure that the cluster maintains efficient operation despite dynamic load and scenario changes, reducing task failure rates due to load fluctuations.

[0020] Finally, through a closed-loop mechanism of data collection, model update, policy iteration, and feedback, the load assessment model parameters and scheduling policy rules can automatically adapt to changes in cluster load characteristics and task distribution. Furthermore, by integrating industry scenario-specific parameters, scheduling policies can be customized for specific scenarios, enhancing the method's versatility and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of the server cluster scheduling method based on dynamic load balancing of the present invention. DETAILED DESCRIPTION

[0022] The following is combined with Figure 1 The present invention is described in further detail: Example 1 See attached Figure 1 As shown, a server cluster scheduling method based on dynamic load balancing includes the following steps: S1: Collect load data of each server in the server cluster in real time. The load data includes static load parameters, dynamic load parameters, and load fluctuation characteristic parameters. S2: A preset load assessment model quantitatively evaluates load data, obtaining the real-time load value and load stability score for each server. Based on the real-time load value, the server is classified into three status levels: idle, normal, and overloaded. The load stability score is used to indicate the severity of server load fluctuations within a preset time period. S3: Receive external tasks to be processed and analyze the resource requirement parameters and task type characteristics of the tasks to be processed; S4: Determine the target server for the task through a preset scheduling strategy based on the server's status level, load stability score, task resource requirement parameters, and task type characteristics; S5: Assign the task to the corresponding target server and monitor the load changes and task progress matching of the target server in real time during the task operation; if the target server load value exceeds the preset overload threshold or the task progress matching is lower than the preset threshold, perform load migration; S6: Based on historical scheduling data, task operation feedback data and industry scenario characteristic parameters, the load assessment model and scheduling strategy are updated in real time.

[0023] The static load parameters are the server's specified hardware inherent attribute parameters, the dynamic load parameters are the performance parameters that change in real time when the server is running, and the load fluctuation characteristic parameters are the fluctuation amplitude, frequency and trend slope of the dynamic load within a preset time period; The task progress matching degree is the deviation rate between the actual task progress and the preset progress; The resource requirement parameters include the CPU usage, memory usage, network bandwidth, and task priority required by the task; the task type characteristics include whether the task is compute-intensive, IO-intensive, or mixed, and the expected runtime of the task; The industry scenario characteristic parameters include the distribution pattern of task types within the scenario, peak period distribution and common characteristics of resource demand.

[0024] The scheduling strategy includes: Prioritize assigning tasks to servers that are idle and have a load stability score ≥ the preset threshold; For computing-intensive tasks, servers with high CPU performance redundancy are prioritized, and for IO-intensive tasks, servers with high disk IO throughput are prioritized. If the task is of high priority and the normal state server has insufficient resources, the task is assigned after triggering the load migration of the overloaded state server.

[0025] The load fluctuation characteristic parameters are calculated by a sliding window algorithm: a sliding window is used in units of 5 minutes, the standard deviation of the dynamic load parameters within the statistical window is used as the fluctuation amplitude, the number of peak occurrences is used as the fluctuation frequency, and the linear fitting slope is used as the fluctuation trend; Among them, when the fluctuation trend has a positive slope, it indicates that the load is on an upward trend, and when it has a negative slope, it indicates that the load is on a downward trend.

[0026] Example 2 The specific process of quantitatively evaluating the load data using the preset load evaluation model in step S2 to obtain the real-time load value and load stability score of each server is as follows: S21: Preprocess the load data of each server, filter outliers in the dynamic load parameters using the 3σ principle, normalize parameters of different units to dimensionless values ​​in the range of 0-1, and align the load fluctuation characteristic parameters with the timestamps of the real-time dynamic load parameters. S22: Preset basic weights based on server type: Static load parameters, including the number of CPU cores and total memory capacity, are assigned a fixed base weight of 30% of the total weight used in the real-time load calculation. This weight is 15% for the number of CPU cores, 10% for total memory capacity, and 5% for the disk storage limit, as disks have a relatively low impact on real-time load. Dynamic load parameters including real-time CPU usage and memory usage: dynamic basic weights are assigned, accounting for 70% of the total weight of the real-time load value calculation. In the initial state, the real-time CPU usage weight is 30%, the real-time memory usage weight is 20%, the network bandwidth usage weight is 15%, and the task queue length weight is 5%. Load fluctuation characteristic parameters including fluctuation amplitude, frequency, and trend slope: corresponding to the basic weights of load stability score of 40%, 30%, and 30% respectively; S23: Calculate the real-time load value to quantify the current load intensity of the server: Static load parameter quantification: Based on the maximum load capacity of the server hardware, the static parameters are converted into quantitative values ​​of hardware redundancy: Static parameter quantization value = (maximum hardware specification - currently allocated specification) / maximum hardware specification; If the server has 32 CPU cores and 8 cores are currently allocated to resident processes, the CPU static redundancy = (32-8) / 32 = 0.75, which means 75% redundancy.

[0027] Dynamic load parameter quantification: Focusing on the server's real-time performance utilization, the weights are dynamically adjusted based on load fluctuation characteristics. Basic quantization: Dynamic parameter quantization value = 1-real-time usage. When the CPU usage is 30%, the quantization value = 0.7; Weight adjustment: If the load fluctuation exceeds the preset threshold, such as a CPU usage fluctuation of more than 20% within 5 minutes, the weight of the dynamic parameter is reduced. For example, the CPU usage weight is reduced from 30% to 20%. At the same time, the weight of the more stable parameter (memory usage) is increased from 20% to 30%. This prevents the evaluation results from being excessively affected by the fluctuating parameters.

[0028] Calculate the real-time load value by weighted sum: Real-time load value = (static parameter quantization value × static weight) + (dynamic parameter quantization value × dynamic corrected weight); When the static redundancy quantization value of a server is 0.8, with a weight of 30%, and the dynamic parameter quantization value is 0.6, with a corrected weight of 70%, the real-time load value = 0.8 × 30% + 0.6 × 70% = 0.66, which is a load intensity of 66%, corresponding to the normal state.

[0029] S24: Calculate the load stability score to quantify the risk of server load fluctuation: Quantification of load fluctuation characteristic parameters: The volatility data calculated based on a 5-minute sliding window is converted into quantitative values: Fluctuation range: Based on the standard deviation of the dynamic load parameter, it is standardized to a score of 0-10. The smaller the standard deviation, the higher the score. For example, if the standard deviation is less than 5%, it will be 10 points; if it is greater than 20%, it will be less than 3 points. Fluctuation frequency: Count the number of times exceeding the preset threshold within 5 minutes, normalized to a score of 0-10, with 10 points for ≤1 and 3 points for ≥5; Trend slope: The load change trend slope is converted into a score of 0-10. If the absolute value of the slope is less than 0.1, it is scored as 10 points; if it is greater than 0.5, it is scored as less than 3 points. This prevents servers that are continuously rising or falling from being judged as stable.

[0030] The stability score is calculated according to the preset weights: Load stability score = (fluctuation amplitude score × 40%) + (fluctuation frequency score × 30%) + (trend slope score × 30%); For example, if a server scores 8 points for fluctuation amplitude, 7 points for fluctuation frequency, and 9 points for trend slope, then the stability score = 8×40%+7×30%+9×30%= 8 points, which is ≥7 points, and is a highly stable server. Example 3 Based on Example 1 or Example 2, the specific process of step S3 is as follows: S31: Perform preliminary verification on the received tasks and filter out invalid or abnormal tasks: Legitimacy verification: Verify whether the identity identifiers carried by the task, including API keys and signatures, match the preset whitelist and reject unauthorized tasks; Completeness check: Checks whether the task contains necessary fields, including the task ID, submission time, and basic resource requirement description. If key fields are missing, an incomplete parameter response is returned and the missing fields are recorded for subsequent user feedback. S32: Based on the resource description information carried by the task, that is, the parameters specified when the task is submitted or the preset task template, extract the four core resource requirements, including CPU requirements: the number of CPU cores or CPU usage upper limit that the parsing task is expected to occupy, such as 2 CPU cores or CPU usage ≤ 50%. If the task is not explicitly specified, the average CPU consumption of similar tasks in the past is used as the basis for estimation. For example, the image compression task is associated with a historical average of 0.5 cores by default.

[0031] Memory requirements: Extract the memory capacity required for the task, distinguishing between initial memory (occupied at task startup) and peak memory (maximum memory occupied during task execution). The latter serves as the core basis for resource matching. Network bandwidth requirements: Determine bandwidth requirements based on whether the task involves data transmission. For example, video transcoding tasks require an upstream bandwidth of ≥10Mbps, while local computing tasks can be set to a lower bandwidth requirement of ≤1Mbps. Storage I / O requirements: For tasks involving disk reads and writes, including database queries and log analysis, analyze the number of I / O operations per second (IOPS) and read and write throughput.

[0032] S33: Task priority analysis and dynamic adjustment: Basic Priority Extraction: This function reads the preset priority level from task parameters, such as P0-P3, with P0 being the highest. If unspecified, the default is P2, the normal priority level. Dynamic Priority Adjustment: This function adjusts the priority level based on task submission time and timeout requirements. For example, a P2 task that has exceeded the timeout by 10 minutes will automatically be promoted to P1, while a P2 task that can be delayed for 24 hours will be downgraded to P3, ensuring that urgent tasks receive resources first.

[0033] S34: Initial task classification based on behavior: Initial task information, including task name, execution instructions, and historical run records, is used for preliminary classification. Computation-intensive tasks: If a task involves a large number of mathematical operations, no frequent disk reads and writes, and minimal network interaction, it is initially judged as computationally intensive and labeled CPU-sensitive.

[0034] IO-intensive tasks: If a task involves a large amount of file reading and writing, database queries, or network data transmission, and has low CPU usage, it is judged to be IO-intensive and marked with the disk IO / network sensitivity label. Hybrid tasks: If both of the above characteristics are met, the task is marked with the CPU+IO dual sensitivity label. Secondary verification of features based on trial runs: For unknown types of tasks that appear for the first time without historical records, a micro-sample trial run mechanism is initiated: extremely small resources of 0.1 CPU cores and 1GB of memory are allocated to the task to run the first 10% of the process, and the resource consumption curve, CPU usage, and IOPS changes during this stage are collected. If the CPU usage during the trial run is continuously ≥60% and IOPS ≤100, it is confirmed to be compute-intensive; if IOPS ≥500 and CPU usage ≤30%, it is confirmed to be IO-intensive, avoiding misjudgments caused by initial information alone. Task duration and correlation analysis: Runtime: Analyze the estimated task execution time or estimate it based on historical data. This timeframe is categorized as short tasks (<10 minutes), medium tasks (10 minutes to 2 hours), and long tasks (>2 hours). Long tasks should be assigned to servers with higher stability to avoid mid-run migrations. Relevance: Check whether the task is part of a task chain. If a dependency exists, record the associated task IDs. During subsequent scheduling, ensure that associated tasks are assigned to servers on the same subnet with a network latency of ≤10ms.

[0035] Example 4 On the basis of Example 1, Example 2, or Example 3, the specific process of matching tasks with target servers in a hierarchical screening manner according to the preset scheduling policy in step S4 is as follows: S41: Filter idle servers and servers with load values ​​≤ 30%: If there is at least one server after screening, select the server ranked first in descending order of comprehensive score (resource matching + type adaptability) as the target server; if there is no idle server that meets the conditions, execute step S42.

[0036] S42: Filter servers in normal state and servers with a load value of 30% < ≤ 70%: If a server that meets the conditions exists after screening, determine the target server in the order of priority adaptability > resource buffer space > stability score; if the normal state server resources are insufficient, for example, there is no matching server for the high priority task, execute step S43; S43: Scheduling of overloaded servers, only for high-priority tasks: If the overloaded server meets the task requirements after migration, it will be determined as the target server; if the migration fails, such as there is no migration target server, or the overloaded server load value is ≥85% and cannot be migrated, the cluster expansion warning will be triggered and emergency scheduling will be initiated.

[0037] The specific process of step S41 is as follows: S411: First perform basic resource matching: Candidate servers that meet the task resource requirement threshold are selected from the available servers. Matching degree calculation: Calculate the matching degree of a single resource by "(server available resources - task requirements) / task requirements". For example, if the server has 3 available CPU cores and the task requires 2, then the CPU matching degree = (3-2) / 2 = 0.5. Select all servers with a resource matching degree ≥ 0, indicating that there is no resource gap. Eliminate invalid candidates: If a server's resource matching degree is less than 0, such as 6GB of available memory less than 8GB of task requirement, it is directly excluded to avoid task failure due to insufficient resources. S412: Task type adaptation: Servers that pass the basic matching are further screened based on task type characteristics. For compute-intensive tasks, servers ranked in the top 30% for CPU redundancy are prioritized (e.g., CPU available cores / total cores ≥ 50%), and servers with CPU fluctuations greater than 20% in the past hour are excluded to avoid impacting computing stability. For I / O-intensive tasks, servers with an IOPS availability of ≥ 60% (i.e., current IOPS usage ≤ 40% of peak) and SSD storage are prioritized. If the task involves cross-node data exchange, servers are additionally required to have network bandwidth redundancy ≥ 1.2 times the task requirements to reserve buffer space. For mixed tasks, a combined score of "CPU matching × 0.4 + I / O matching × 0.4 + stability score × 0.2" is calculated, and servers in the top 50% are selected. S413: Adapting stability and task duration: Optimize selection based on load stability scores and task duration: For long tasks, select only servers with a stability score of 7 or higher to avoid migration caused by load fluctuations. If there are insufficient candidate servers, reduce the score to 6 or higher, but trigger a load fluctuation warning simultaneously. For short tasks, relax the stability requirement (a score of 5 or higher is sufficient) and prioritize servers with the highest resource matching, eliminating the need to wait for highly stable servers to release resources.

[0038] The specific process of step S42 is as follows: S421: Strict verification of resource redundancy: In normal state, servers need to reserve more buffer resources to avoid becoming overloaded after scheduling, so resource matching standards are tightened. Core rule: The server's available resources must be ≥ 1.2 times the task's requirements. For example, if a task requires a 2-core CPU, the server's available CPU must be ≥ 2.4 cores. Special verification: If the task is high priority (P0 / P1), the server's load trend slope (load fluctuation characteristic parameter) is additionally checked. If the slope is > 0.3 (consistently increasing load), the task is excluded even if resources are matched to avoid server overload during task execution. Task priority and server load balancing adaptation: High-priority tasks: Skip the comprehensive score sorting and directly select the server with a resource match ≥ 0.8 and the lowest current load. If two servers meet the requirements, prioritize the one with a load of 35% over 50% to reserve more load headroom. Normal-priority tasks: Calculate the server's current load plus the expected increase in the task's load. For example, if the server's current load is 40% and the task's load is expected to increase by 20%, the total is 60%. Select the server with a total of ≤ 65%, leaving a 5% buffer. Secondary adaptation based on type characteristics: The load stability score of normal servers is used differently. If the task is I / O-intensive and requires frequent disk reads and writes, select a server with a Fluctuation Frequency score of 8 or higher in the stability score to prevent sudden disk I / O fluctuations from impacting the task. If the task is compute-intensive and requires continuous CPU usage, select a server with a Trend Slope score of 7 or higher in the stability score to avoid slowdowns caused by continuously increasing CPU load. The specific process of step S43 is as follows: S431: Overloaded Server Screening and Load Migration Preparation: Screening criteria: Only select servers that are overloaded but not at their limit, with a load of 70%-85%, no automatic protection mechanisms triggered, and server hardware specifications ≥ 1.5 times the task requirements to ensure sufficient resources after the migration. Migration target determination: Select nodes from idle / normal servers that can undertake the migration task. These nodes must meet the migration task resource requirements plus a current load of ≤ 70%. Prioritize servers on the same subnet as the overloaded servers (with network latency ≤ 5ms).

[0039] S432: Load Migration Execution and Task Allocation: Migration task selection: Based on the principle of low priority and low migration cost, such as tasks with a priority of P3 and a migration time of less than 30 seconds, calculate the total number of tasks to be migrated and ensure that the load value of the overloaded server drops to ≤ 60%. Task assignment timing: After the migration is completed (heartbeat monitoring confirms that the migration task is running normally), immediately assign the current high-priority task to the server and mark it in temporary protection status. No new tasks will be assigned for 3 minutes to prevent further overload.

[0040] The specific process of step S6 is: first, trigger the update according to events such as light updates every 30 minutes, full updates every day, or too high task failure rate. Then, collect relevant data, and for the load assessment model, adjust the weights of static / dynamic load parameters by analyzing the deviation between the model prediction value and the actual value. For example, if the prediction deviation of IO-intensive tasks is large, increase the IO parameter weight. For the scheduling strategy, optimize the rules based on task efficiency feedback, such as increasing the allocation weight of IO-intensive tasks on SSD servers. Finally, perform grayscale verification after the update, and fully deploy if there is no problem. At the same time, record the update log to form a data feedback closed loop.

Claims

1. A server cluster scheduling method based on dynamic load balancing, characterized in that: The following steps are involved: S1: Collect load data of each server in the server cluster in real time. The load data includes static load parameters, dynamic load parameters, and load fluctuation characteristic parameters. S2: A preset load assessment model quantitatively evaluates load data, obtaining the real-time load value and load stability score for each server. Based on the real-time load value, the server is classified into three status levels: idle, normal, and overloaded. The load stability score is used to indicate the severity of server load fluctuations within a preset time period. S3: Receive external tasks to be processed and analyze the resource requirement parameters and task type characteristics of the tasks to be processed; S4: Determine the target server for the task through a preset scheduling strategy based on the server's status level, load stability score, task resource requirement parameters, and task type characteristics; S5: Assign tasks to the corresponding target servers and monitor the load changes and task progress matching of the target servers in real time during the task execution. If the target server load value exceeds the preset overload threshold or the task progress matching degree is lower than the preset threshold, load migration is performed; S6: Based on historical scheduling data, task operation feedback data and industry scenario characteristic parameters, the load assessment model and scheduling strategy are updated in real time.

2. A server cluster scheduling method based on dynamic load balancing according to claim 1, characterized in that: The static load parameters are the server's specified hardware inherent attribute parameters, the dynamic load parameters are the performance parameters that change in real time when the server is running, and the load fluctuation characteristic parameters are the fluctuation amplitude, frequency and trend slope of the dynamic load within a preset time period; The task progress matching degree is the deviation rate between the actual task progress and the preset progress; The resource requirement parameters include the CPU usage, memory usage, network bandwidth, and task priority required by the task; the task type characteristics include whether the task is compute-intensive, IO-intensive, or mixed, and the expected runtime of the task; The industry scenario characteristic parameters include the distribution pattern of task types within the scenario, peak period distribution and common characteristics of resource demand.

3. A server cluster scheduling method based on dynamic load balancing according to claim 1, characterized in that: The scheduling strategy includes: Prioritize assigning tasks to servers that are idle and have a load stability score ≥ the preset threshold; For computing-intensive tasks, servers with high CPU performance redundancy are prioritized, and for IO-intensive tasks, servers with high disk IO throughput are prioritized. If the task is of high priority and the normal state server has insufficient resources, the task is assigned after triggering the load migration of the overloaded state server.

4. A server cluster scheduling method based on dynamic load balancing according to claim 2, characterized in that: The load fluctuation characteristic parameters are calculated using a sliding window algorithm: a sliding window is used in units of 5 minutes, the standard deviation of the dynamic load parameters within the statistical window is used as the fluctuation amplitude, the number of peak occurrences is used as the fluctuation frequency, and the linear fitting slope is used as the fluctuation trend; wherein, a positive slope indicates an upward trend in the load, and a negative slope indicates a downward trend in the load.

5. The server cluster scheduling method based on dynamic load balancing according to claim 1, characterized in that: The specific process of quantitatively evaluating the load data using the preset load evaluation model in step S2 to obtain the real-time load value and load stability score of each server is as follows: S21: Preprocessing the load data of each server; S22: Preset basic weights based on server type: Assign fixed basic weights to static load parameters including the number of CPU cores and total memory capacity; Assign dynamic basic weights to dynamic load parameters including real-time CPU usage and memory usage; The load fluctuation characteristic parameters including fluctuation amplitude, frequency and trend slope are respectively assigned basic weights for the load stability score; S23: Calculate the real-time load value to quantify the current load intensity of the server: Static load parameter quantification: Based on the maximum load capacity of the server hardware, the static parameters are converted into a quantitative value of hardware redundancy: Static parameter quantification value = (maximum hardware specification - currently allocated specification) / maximum hardware specification; Dynamic load parameter quantification: Focusing on the server's real-time performance utilization, the weights are dynamically adjusted based on load fluctuation characteristics. Basic quantization: dynamic parameter quantization value = 1-real-time usage rate; Weight correction: If the load fluctuation range is greater than the preset threshold, the weight of the dynamic parameter is reduced, while the weight of the parameter with higher stability is increased; Calculate the real-time load value by weighted sum: Real-time load value = (static parameter quantization value × static weight) + (dynamic parameter quantization value × dynamic corrected weight); S24: Calculate the load stability score to quantify the risk of server load fluctuation: Quantification of load fluctuation characteristic parameters: The volatility data calculated based on a 5-minute sliding window is converted into quantitative values: Fluctuation amplitude: based on the standard deviation of the dynamic load parameter, standardized to a score of 0-10; Fluctuation frequency: counts the number of times exceeding the preset threshold within 5 minutes, normalized to a score of 0-10; Trend slope: Convert the load change trend slope into 0-10 points; The stability score is calculated according to the preset weights: Load stability score = (fluctuation amplitude score × 40%) + (fluctuation frequency score × 30%) + (trend slope score × 30%).

6. A server cluster scheduling method based on dynamic load balancing according to claim 1, characterized in that: The specific process of step S3 is as follows: S31: Perform preliminary verification on the received tasks to filter out invalid or abnormal tasks, including legality verification and integrity verification; S32: Based on the resource description information carried by the task, that is, the parameters specified when the task is submitted or the preset task template, extract the four core resource requirements, including CPU requirements, memory requirements, network bandwidth requirements, and storage I / O requirements; S33: Task priority analysis and dynamic adjustment: Basic priority extraction: read the preset priority levels P0-P3 from the task parameters; Dynamic priority adjustment: Adjustment based on task submission time and timeout requirements; S34: Initial judgment based on task behavior type: Preliminary classification based on initial task information: Computationally intensive tasks: If the task involves a large amount of mathematical operations, no frequent disk reads and writes, and little network interaction, it is initially judged to be computationally intensive and marked with a CPU-sensitive label; I / O-intensive tasks: If a task involves a large amount of file reading and writing, database queries, or network data transmission, and the CPU usage is lower than the preset value, it is determined to be I / O-intensive and marked with a disk I / O / network sensitive label; Hybrid tasks: If they meet the characteristics of both computation-intensive and computation-intensive tasks, they will be marked with CPU+IO dual-sensitive labels; Secondary verification of features based on trial runs: For the first appearance of an unknown type of task, a micro-sample trial run mechanism is initiated: minimal resources are allocated to allow the task to run the first 10% of the process, and the resource consumption curve of this stage is collected; Task duration and correlation analysis: Runtime: Analyze the estimated execution time of the task or estimate it based on historical data, and categorize the duration into short tasks, medium tasks, and long tasks; Relevance: Check whether the task is part of a task chain. If there is a dependency, record the associated task ID. During subsequent scheduling, ensure that the associated tasks are assigned to the same subnet server with a network latency of ≤10ms.

7. A server cluster scheduling method based on dynamic load balancing according to claim 1, characterized in that: The specific process of matching tasks with target servers in a hierarchical screening manner according to the preset scheduling policy in step S4 is as follows: S41: Filter idle servers, servers with load values ​​≤ 30%: If there is at least one server after screening, the server with the highest ranking will be selected as the target server in descending order of the comprehensive score of resource matching degree + type adaptability; If there is no idle server that meets the conditions, execute step S42; S42: Filter servers in normal status, servers with a load value of 30% < ≤ 70%: If there are servers that meet the conditions after screening, the target server is determined in the order of priority adaptability > resource buffer space > stability score; if the server resources are insufficient in the normal state, such as there is no matching server for the high-priority task, step S43 is executed; S43: Overloaded server scheduling, only for high-priority tasks: If the overloaded server meets the task requirements after migration, it will be determined as the target server; if the migration fails, such as there is no migration target server, or the overloaded server load value is ≥85% and cannot be migrated, the cluster expansion warning will be triggered and emergency scheduling will be initiated.

8. A server cluster scheduling method based on dynamic load balancing according to claim 7, characterized in that: The specific process of step S41 is as follows: S411: First perform basic resource matching: Filter candidate servers that meet the task resource requirement threshold from idle servers: Matching degree calculation: Calculate the matching degree of a single resource by "(server available resources - task requirements) / task requirements"; Eliminate invalid candidates: If the matching degree of a certain resource on the server is less than 0, it will be directly excluded; S412: Perform task type adaptation: From the servers that have passed the basic matching, further filter by task type characteristics: Computation-intensive tasks: Prioritize servers ranked in the top 30% in terms of CPU redundancy, and exclude servers with CPU fluctuations greater than 20% in the past hour. IO-intensive tasks: Prioritize IOPS availability ≥ 60%; Hybrid tasks: Calculate the combined score of "CPU matching × 0.4 + I / O matching × 0.4 + Stability score × 0.2" and select the top 50% of servers. S413: Adapting stability and task duration: Combined with the load stability score and task duration to optimize the selection: Long missions longer than 2 hours: only select servers with a stability score of 7 or higher. Short tasks shorter than 10 minutes: Select from servers with a stability score of 5 or higher, prioritizing servers with the highest resource match. There is no need to wait for highly stable servers to release resources.

9. The server cluster scheduling method based on dynamic load balancing according to claim 7, characterized in that: The specific process of step S42 is as follows: S421: Strict verification of resource redundancy: In normal state, servers need to reserve more buffer resources to avoid being overloaded after scheduling. Therefore, the resource matching standards are improved: Core rules: Server available resources must be ≥ 1.2 times the task requirements; Special check: If the task is of high priority, P0 / P1, the server load trend slope is additionally checked. If the slope is greater than 0.3, the task is excluded even if resources are matched. S422: Task priority and server load balancing adaptation: High-priority tasks: Skip the comprehensive score sorting and directly select the server with a resource matching degree ≥ 0.8 and the lowest current load value; Normal priority tasks: calculate the current server load value + the expected increase in load value of the task; Type feature secondary adaptation: S423: Differentiated use of load stability scores for servers in normal state: If the task is I / O-intensive and requires frequent disk reads and writes, select a server with a stability score of 8 or higher. If the task is computationally intensive and requires continuous CPU usage, select a server with a trend slope score of 7 or higher in the stability score.

10. The server cluster scheduling method based on dynamic load balancing according to claim 7, characterized in that: The specific process of step S43 is as follows: S431: Overloaded Server Screening and Load Migration Preparation: Filtering criteria: Only servers that are overloaded but not at their limit are selected, with a load value of 70%-85%, no automatic protection mechanism triggered, and server hardware specifications ≥ 1.5 times the task requirements; Migration target determination: Select nodes that can undertake the migration task from idle / normal servers. The nodes must meet the migration task resource requirements and the current load must be ≤ 70%. Prioritize servers in the same subnet as the overloaded servers. S432: Load Migration Execution and Task Allocation: Migration task selection: Based on the principles of low priority and low migration cost; Task assignment timing: After the migration is completed, the current high-priority task will be immediately assigned to the server and marked as temporarily protected. No new tasks will be assigned within the specified time.

Images