Spark system resource utilization rate sensing task scheduling method based on boar genetic evaluation calculation characteristics
By using a sliding window-based node performance evaluation and a roulette wheel strategy to select low-performance nodes, combined with weighted random sorting and task migration scheduling, the resource management and network congestion problems in the genetic evaluation of breeding pigs in heterogeneous Spark systems are solved, thereby improving system performance and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHANGSHA UNIVERSITY OF SCIENCE AND TECHNOLOGY
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-28
AI Technical Summary
Existing heterogeneous Spark systems cannot efficiently adapt to computational characteristics in pig genetic evaluation tasks, resulting in low resource management and scheduling efficiency, neglect of dynamic differences in node performance, uneven resource utilization, network congestion, and other problems that affect the efficiency of genetic evaluation.
A real-time node performance evaluation method based on sliding window is adopted. A roulette wheel strategy is used to select low-performance nodes with idle network. Combined with weighted random sorting and task migration or pre-execution scheduling, task allocation is optimized to achieve resource utilization-aware task scheduling.
It improves task execution efficiency, enhances CPU and network resource utilization, shortens the completion time of genetic evaluation jobs, outperforms existing scheduling strategies, and achieves a balance between computational and network loads.
Smart Images
Figure BDA0005774062150000021 
Figure BDA0005774062150000031 
Figure BDA0005774062150000036
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent genetic evaluation system software for whole-genome selection and task scheduling technology for parallel distributed computing. It relates to a resource utilization-aware task scheduling method for the Spark system based on the computational characteristics of bovine genetic evaluation. Background Technology
[0002] With the exponential growth of genomic data and the advancement of artificial intelligence technology, modern pig genetic breeding can leverage AI tools such as machine learning and neural networks to analyze massive amounts of data, thereby accurately predicting genotype-phenotype associations and improving the accuracy of pig genetic assessment. However, pig whole-genome selection information is characterized by low value density, large scale, complex data associations, and high multimodal dimensionality, posing challenges to the efficiency of pig genetic assessment. In recent years, distributed big data processing platforms based on heterogeneous Spark systems have been widely used in the field of bio-breeding. However, existing heterogeneous Spark systems cannot efficiently adapt to the computational characteristics of pig genetic assessment tasks, resulting in low system resource management and scheduling efficiency. Therefore, proposing a Spark task scheduling method suitable for the computational characteristics of pig genetic assessment is an important means to improve its system performance.
[0003] Existing distributed task scheduling methods are typically based on the principle of prioritizing "data locality," that is, allocating computational tasks to nodes that store the corresponding data for execution as much as possible to reduce network transmission overhead. Their scheduling process is as follows: Figure 1 As shown. However, in heterogeneous cluster environments oriented towards computational characteristics for breeding pig genetic evaluation, this static or location-aware scheduling method has significant drawbacks, specifically:
[0004] 1. Ignoring the dynamic differences in node performance: When performing large-scale genetic evaluation calculations, low-performance nodes often become bottlenecks due to insufficient computing power, which greatly prolongs the completion time of the entire evaluation task.
[0005] 2. Uneven resource utilization and computational starvation: High-performance nodes, due to their faster processing speed, often complete local tasks earlier than low-performance nodes. Under the default policy, in pursuit of data locality, high-performance nodes may be idle and waiting, resulting in a waste of expensive computing resources; in addition, when triggering delayed scheduling, the network status is not fully considered, and remote data is blindly fetched.
[0006] 3. Performance degradation due to network congestion: Swine genome data is typically high-dimensional and dense (e.g., a matrix of hundreds of thousands of SNP loci from tens of thousands of pigs). When high-performance nodes pull large amounts of data from low-performance nodes for computation, a lack of awareness and control over network bandwidth can easily lead to sudden network congestion. In this case, the data transmission time may outweigh the benefits of computational acceleration.
[0007] This reduced the overall throughput of the system.
[0008] In summary, in the intelligent genetic evaluation scenario of whole-genome selection in breeding pigs, facing heterogeneous computing resources and massive multimodal data, how to design a method that can perceive node resource utilization in real time, balance computing and network load, and dynamically optimize task allocation has become a key technical problem that urgently needs to be solved to improve the efficiency of genetic evaluation and shorten the breeding cycle. Summary of the Invention
[0009] The technical problem to be solved by the present invention is to overcome the above-mentioned defects of the prior art and provide a Spark system resource utilization awareness task scheduling method based on the genetic evaluation calculation characteristics of breeding pigs.
[0010] A resource utilization-aware task scheduling method for Spark systems based on the computational characteristics of breeding pig genetic evaluation includes the following steps: S100: Evaluate the performance of heterogeneous nodes in real time using a sliding window based on the historical task completion status of nodes; S200: Obtain low-performance nodes with idle networks and select task-providing nodes using a roulette wheel strategy; S300: Obtain an ordered list of task-receiving nodes using weighted random sorting; S400: Execute a task migration or pre-execution scheduling scheme based on the network status and data distribution of the task-receiving nodes; S500: Update the task list and the network status of the nodes.
[0011] Step S100 includes: S101: Setting the host of the idle actuator as the base host h base and its performance value hp base Set the unit standard value to 1; S102: Select the most recently completed γ tasks from the historical completed task queue of the benchmark host as the benchmark task set (i.e., sliding window), and calculate the average processing rate of the benchmark task set, denoted as the benchmark rate v. base , where d i , t i These represent the data volume and completion time of the i-th task, respectively.
[0012]
[0013] S103: From the host to be evaluated h j From the historical completed task queue, the γ most recently completed tasks are selected as the comparison task level. The average processing rate of the comparison task set is calculated and denoted as the comparison rate v. j The ratio of the comparison rate to the reference rate is determined as the relative performance value of the host to be evaluated relative to the reference host, and is recorded as the performance value (hp) of the host to be evaluated. j .
[0014]
[0015] Step S200 includes: S201: Based on node h j Network blocking deadline B j With current system time Determine the network state based on the relationship; if the condition is satisfied... Then the node is determined to be in a network idle state; B j The calculation method is as follows, where This indicates the blocking time caused by the previous round of task transmission. Indicates the previous round h j The time T for updating network status com Indicates the preset compensation time;
[0016]
[0017] S202: Based on the node performance values evaluated in step S100, nodes with performance values lower than a preset low-performance threshold are marked as low-performance nodes; all nodes in the cluster are traversed, and nodes that simultaneously meet the criteria of being marked as low-performance nodes and are in an idle network state are selected to form a candidate set of task-providing nodes (FLS); S203: Weights are assigned to the nodes in the FLS: if there is a node with a performance value of 0, it is assigned a preset fixed weight value (e.g., 1.0) and other nodes are removed; if there is no node, the node is assigned a weight value that is the reciprocal of its performance. S204: The total weight W of the candidate set is calculated. total and in the interval [0, W total A random threshold R is generated within the candidate set; the order of nodes in the candidate set is randomly shuffled, and the weights of each node are accumulated one by one, selecting the accumulated weight value W. c First satisfaction of W c Nodes ≥R serve as task-providing nodes h pro .
[0018] Step S300 includes: S301: Based on the performance evaluation results in step S100, select nodes with performance values higher than a preset high-performance threshold to form a candidate set HS for task receiving nodes; S302: For each node h in HS... j Calculate the sort value x j Specifically, a uniformly distributed random variable U is generated within the interval [0,1], and the sort value of the node is calculated using the following formula, where -logU transforms the uniform distribution into an exponential distribution.
[0019]
[0020] S303: Sort the candidate nodes in ascending order according to the sorting value described in S302, and generate an ordered list CS of task receiving nodes.rec This is to ensure that nodes with higher performance values generally have a higher probability of priority reception.
[0021] Step S400 includes: S401: Traversing the task partition list of the task providing node described in step S204, and selecting the first unscheduled and unmigrated task partition as the target task; S402: Traversing the candidate nodes h in the ordered list of task receiving nodes generated in step S303 in sequence. j If h j If the data partition containing the target task is present, the pre-execution mechanism is triggered: directly in h j Calculate the task above and immediately end the current scheduling process; if h j If no data partition exists, but the network idle state described in step S201 is met, then the task will be migrated (data transferred) to h. j Run the process and end the current scheduling process; if the network is not idle, continue the traversal.
[0022] Step S500 includes: marking the target task mentioned in S401 as scheduled; if the task is run by migration, updating the network status of the migration nodes.
[0023] This invention provides a Spark system resource utilization-aware task scheduling method based on the genetic evaluation calculation characteristics of breeding pigs. This invention is of great significance for improving task execution efficiency and increasing the utilization of CPU and network resources.
[0024] This invention addresses the significant performance differences among heterogeneous computing nodes, the dynamic fluctuations in node computing power over time, and the low resource utilization and network congestion caused by existing scheduling methods neglecting network conditions in the processing of massive data related to the selection of the entire genomic genome of breeding pigs. It proposes a resource utilization-aware task scheduling method for Spark systems based on the computational characteristics of genomic evaluation of breeding pigs. This method effectively achieves real-time evaluation and accurate quantification of node performance, fully utilizes idle bandwidth and high computing power resources while ensuring data locality, and achieves a dual balance between computational and network load, thus accelerating the processing speed of genetic evaluation jobs. Experimental results demonstrate that this method significantly outperforms Spark's default delayed scheduling strategy and existing speculative execution mechanisms in improving CPU and network utilization and shortening job completion time.
[0025] To verify the effectiveness of the Spark system resource utilization-aware task scheduling method based on the computational characteristics of breeding pig genetic evaluation proposed in this invention in the scenario of big data processing for whole-genome selection of breeding pigs, this embodiment constructs a real heterogeneous cluster environment. The cluster contains 8 physical nodes, all located on the same rack, with a network bandwidth of 1Gbps. The node configuration exhibits significant heterogeneity, specifically as follows: High-performance nodes (3 units): configured with 20-32 CPU cores and 32GB of memory, representing the latest high-computing servers purchased by the breeding center; Medium-performance nodes (3 units): configured with 8-12 CPU cores and 16GB of memory; Low-performance nodes (2 units): configured with 8 CPU cores and 48GB of memory, representing older servers. To demonstrate the superiority of this invention, the method is compared with the following three existing mainstream scheduling methods: SDEFA: Spark's default delayed scheduling method; SSPEC: SDEFA combined with speculative execution. Figure 4 As shown, compared to SDEFA and SSPEC, the method of this invention reduces application completion time by an average of 23.74% and 11.99%, respectively. Regarding resource utilization, as... Figure 5 The present invention can effectively avoid the task waiting process in which the CPU utilization of SDEFA and SSPEC drops significantly and then rises again in the later stage of application operation; in addition, the present invention can also effectively utilize idle bandwidth. Attached Figure Description
[0026] Figure 1 This refers to existing technology-based task scheduling solutions;
[0027] Figure 2 This is the system architecture of the present invention;
[0028] Figure 3 This is the task scheduling scheme of the present invention;
[0029] Figure 4 This is a comparison chart of the completion time of the present invention under WordCount, LDA and LR applications respectively.
[0030] Figure 5 This is a comparison chart of the CPU and network load of the present invention. Detailed Implementation
[0031] This embodiment implements a resource utilization-aware task scheduling method for Spark systems based on the computational characteristics of breeding pig genetic evaluation. This method primarily runs on the Driver component of the Spark cluster and includes the following core modules: an information collector responsible for collecting historical task information from each worker node in the cluster in real time; a performance evaluator used to quantify the real-time processing capacity of each node based on historical task completion; and a task scheduler, the core decision-making module, which includes logic for task provider selection (S200), task receiver sorting (S300), and migration / pre-execution decision (S400).
[0032] The main process of the scheduling method (DRUTS) proposed in this invention is as follows: Figure 3 As shown, the following steps will be implemented in detail:
[0033] S100 (Real-time Performance Evaluation of Heterogeneous Nodes Based on Sliding Window): To address the issue of performance fluctuations in heterogeneous environments, this embodiment employs relative performance metrics. The specific implementation logic is as follows:
[0034] S101 (Baseline Setting): When a task is completed, the executor releases an idle core and uses its host as the base host h. host Set its baseline performance value h host =1.0.
[0035] S102 (Sliding Window Calculation): Set the sliding window size γ (e.g., select the 4 most recently completed tasks); select the γ most recently completed tasks from the historical completed task queue of the baseline host as the baseline task set, and calculate the average processing rate of the baseline task set, denoted as the baseline rate v. base , where d i , t i Let represent the data volume and completion time of the i-th task, respectively.
[0036]
[0037] S103 (Relative Performance Quantization): For the host h to be evaluated j Similarly, the average rate v is calculated by selecting the γ most recently completed tasks. j The node's real-time performance value (hp) j The calculation formula is:
[0038]
[0039] S200 (Task Provider Node Filtering): To avoid the bottleneck effect in computation, tasks on low-performance nodes need to be moved out. The task provider node logic is as follows:
[0040] S201 (Network Status Judgment): Based on node hj Network blocking deadline B j With current system time Determine the network state based on the relationship; if the condition is satisfied... Then the node is determined to be in a network idle state; B j The calculation method is as follows, where This indicates the blocking time caused by the previous round of task transmission. Indicates the previous round h j The time T for updating network status com Indicates the preset compensation time;
[0041]
[0042] S202 (Candidate Set Construction): Traverse the nodes and select nodes that simultaneously meet the following two conditions to form a candidate set FLS: performance value hp j Below a preset low performance threshold (e.g., 0.67); in an idle network state.
[0043] S203 (Roulette Weight Assignment): Assign weights to the nodes in the FLS: If there is a node with a performance value of 0 (i.e., a node that completes 0 tasks, which is considered the lowest performance node and should be given priority), assign it a preset fixed weight value (e.g., 1.0) and remove other nodes; if there is no node, assign a weight value to the node as the reciprocal of its performance, i.e., the lower the performance, the greater the probability of being selected.
[0044] S204 (Roulette Selection): Calculate the sum of weights W of the candidate set. total and in the interval [0, W total A random threshold R is generated within the candidate set; the order of nodes in the candidate set is randomly shuffled, and the weights of each node are accumulated one by one, selecting the accumulated weight value W. c First satisfaction of W c Nodes ≥R serve as task-providing nodes h pro .
[0045] S300 (Weighted random sorting of the candidate set of task receiving nodes): The purpose of this step is to distribute tasks to more capable nodes while maintaining load balancing.
[0046] S301 (Filtering high-performance nodes): Based on the performance results evaluated in step S100, nodes with performance values higher than the preset high-performance threshold are filtered to form a candidate set HS for task receiving nodes.
[0047] S302 (Calculate the sorting value): For each node h in HS j Calculate the sort value x jSpecifically, a uniformly distributed random variable U is generated within the interval [0,1], and the ranking value of the node is calculated using the following formula, where -logU transforms the uniform distribution into an exponential distribution. The exponential distribution has a larger variance than the uniform distribution, which can highlight the influence of randomness on node selection when there are significant differences in node performance.
[0048]
[0049] S303 (Generate List): Arrange the candidate nodes in ascending order according to the sorting value described in S302, and generate an ordered list CS of task receiving nodes. rec This is to ensure that nodes with higher performance values generally have a higher probability of priority reception.
[0050] S400 (Task Migration or Pre-Execution Scheduling): This is the core decision-making step in scheduling, determining the task transfer method based on the data location. The specific implementation logic is as follows:
[0051] S401 (Get the target task): Traverse the task partition list of the task providing node described in step S204, and select the first unscheduled and unmigrated task partition as the target task;
[0052] S402 (Transfer Task): Iterate through the candidate nodes h in the ordered list of task receiving nodes generated in step S303. j If h j If the data partition containing the target task is present, the pre-execution mechanism is triggered: directly in h j Calculate the task above and immediately end the current scheduling process; if h j If no data partition exists, but the network idle state described in step S201 is met, then the task will be migrated (data transferred) to h. j Run the process and end the current scheduling process; if the network is not idle, continue the traversal.
[0053] S500 (Update Task and Network Status): Mark the target task described in S401 as scheduled; if the task is run via migration, update the network status of both migration nodes.
[0054] This embodiment compares the performance of the proposed method (DRUTS) with Spark's default delayed scheduling (SDEFA) and the combined speculative execution strategy (SSPEC) using three typical workloads (WordCount, LDA, LR) in the context of whole-genome selection in breeding pigs (GBLUP). Figure 4As shown, compared to SDEFA and SSPEC, the DRUTS strategy reduced the average completion time of applications by 23.74% and 11.99%, respectively, indicating that the present invention can effectively eliminate the long-tail effect of the cluster. Regarding resource utilization, Figure 5 As shown, the SDEFA strategy in the later stages of job execution suffers from a significant "trough" (idle period) in CPU utilization due to waiting for data locality or slow nodes. In contrast, DRUTS, through proactive scheduling, maintains a stable high level of CPU utilization and shortens the task "tailing" time. Figure 5 As shown, DRUTS can sense network status. When the network is idle (during the trough of the curve), it will make appropriate use of bandwidth to migrate tasks (the DRUTS down / up curve shows obvious activity) in exchange for faster computing speed; while when the network is busy, it will automatically avoid it, thus achieving dual load balancing of computing and transmission.
[0055] Many specific details have been set forth in the foregoing description to provide a thorough understanding of the present invention. However, the above description is merely a preferred embodiment of the present invention, and the present invention can be implemented in many other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed above. Furthermore, any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention, or modify them into equivalent embodiments, using the methods and techniques disclosed above, without departing from the scope of the present invention. Any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention, without departing from the content of the present invention, shall still fall within the protection scope of the present invention.
Claims
1. A Spark system resource utilization-aware task scheduling method based on the computational characteristics of breeding pig genetic evaluation, comprising the following steps: S100: Based on the historical task completion status of nodes, use a sliding window to evaluate the performance of heterogeneous nodes in real time; S200: Obtain low-performance nodes with idle networks, and use a roulette wheel strategy to select the node that provides the task; S300: Obtain an ordered list of task receiving nodes using weighted random sorting; S400: Executes task migration or pre-execution scheduling schemes based on the network status and data distribution of the task receiving node; S500: Update the task list and the network status of nodes.
2. The Spark system resource utilization-aware task scheduling method based on the genetic evaluation calculation characteristics of breeding pigs according to claim 1, characterized in that, Step S100 includes: S101: Set the host machine where the idle actuator is located as the reference host h base and its performance value hp base Set the unit standard value to 1; S102: Select the γ most recently completed tasks from the historical completed task queue of the benchmark host as the benchmark task set (i.e., the sliding window), and calculate the average processing rate of the benchmark task set, denoted as the benchmark rate v. base , where d i , t i These represent the data volume and completion time of the i-th task, respectively. S103: From the host to be evaluated h j From the historical completed task queue, the γ most recently completed tasks are selected as the comparison task level. The average processing rate of the comparison task set is calculated and denoted as the comparison rate v. j The ratio of the comparison rate to the reference rate is determined as the relative performance value of the host to be evaluated relative to the reference host, and is recorded as the performance value (hp) of the host to be evaluated. j .
3. The Spark system resource utilization awareness task scheduling method based on the genetic evaluation calculation characteristics of breeding pigs according to claim 2, characterized in that, Step S200 includes: S201: Based on node h j Network blocking deadline B j With current system time Determine the network state based on the relationship; if the condition is satisfied... Then the node is determined to be in a network idle state; B j The calculation method is as follows, where This indicates the blocking time caused by the previous round of task transmission. Indicates the previous round h j The time T for updating network status com Indicates the preset compensation time; S202: Based on the node performance value evaluated in step S100, nodes with performance values lower than the preset low performance threshold are marked as low performance nodes; all nodes in the cluster are traversed, and nodes that simultaneously meet the criteria of being marked as low performance nodes and are in an idle network state are selected to form a task-providing node candidate set FLS. S203: Assign weights to the nodes in the FLS: If there is a node with a performance value of 0, assign it a preset fixed weight value (e.g., 1.0) and remove other nodes; if there is no node, assign the node a weight value that is the reciprocal of its performance. S204: Calculate the sum of weights W of the candidate set total and in the interval [0, W total A random threshold R is generated within the candidate set; the order of nodes in the candidate set is randomly shuffled, and the weights of each node are accumulated one by one, selecting the accumulated weight value W. c First satisfaction of W c Nodes with a value ≥ R serve as task-providing nodes h pro .
4. The Spark system resource utilization awareness task scheduling method based on the genetic evaluation calculation characteristics of breeding pigs according to claim 3, characterized in that, Step S300 includes: S301: Based on the performance results evaluated in step S100, select nodes with performance values higher than the preset high performance threshold to form a candidate set HS for task receiving nodes. S302: For each node h in HS j Calculate the sort value x j Specifically, a uniformly distributed random variable U is generated within the interval [0,1], and the sort value of the node is calculated using the following formula, where -logU transforms the uniform distribution into an exponential distribution. S303: Sort the candidate nodes in ascending order according to the sorting value described in S302, and generate an ordered list CS of task receiving nodes. rec This is to ensure that nodes with higher performance values have a higher probability of being prioritized for reception.
5. The Spark system resource utilization awareness task scheduling method based on the genetic evaluation calculation characteristics of breeding pigs according to claim 4, characterized in that, Step S400 includes: S401: Traverse the task partition list of the task providing node described in step S204, and select the first unscheduled and unmigrated task partition as the target task; S402: Traverse the candidate nodes h in the ordered list of task receiving nodes generated in step S303 in sequence. j If h j If the data partition containing the target task is present, the pre-execution mechanism is triggered: directly in h j Calculate the task above and immediately end the current scheduling process; if h j If no data partition exists, but the network idle state described in step S201 is met, then the task will be migrated (data transferred) to h. j Run the process and end the current scheduling process; if the network is not idle, continue the traversal.
6. The Spark system resource utilization-aware task scheduling method based on the genetic evaluation calculation characteristics of breeding pigs according to claim 5, characterized in that, Step S500 includes: marking the target task mentioned in S401 as scheduled; if the task is run by migration, updating the network status of the migration nodes.