Distributed computing task scheduling optimization method and system for tubercle bacillus research data

Abstract

The invention relates to the technical field of distributed computing, and discloses a distributed computing task scheduling optimization method and system for tubercle bacillus research data, and the method comprises the steps: extracting the CPU intensity, the memory demand, the I / O characteristic and the data dependence characteristic vector of a computing task; constructing a task dependence directed acyclic graph; determining a task execution sequence through topological sorting and key path analysis; constructing a multi-dimensional fitness matrix of task-resource matching; determining a task allocation scheme by adopting a scheduling strategy combining dynamic planning and heuristic search; predicting a resource demand through an adaptive resource reservation mechanism and dynamically adjusting the resource reservation amount; according to the method, collaborative optimization of accurate perception of task features and intelligent matching of resources is realized, and the resource utilization rate and the task processing efficiency are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of distributed computing and task scheduling technology, specifically to a method and system for optimizing distributed computing task scheduling for Mycobacterium tuberculosis research data. Background Technology

[0002] With the deepening of tuberculosis prevention and control research, research data related to Mycobacterium tuberculosis is experiencing explosive growth. The scale of genome sequencing data, protein structure data, drug resistance analysis data, and epidemiological survey data continues to expand, leading to an increasing demand for computing resources. Distributed computing clusters have become an important infrastructure for processing large-scale Mycobacterium tuberculosis research data, and task scheduling optimization directly affects the utilization efficiency of computing resources and the timeliness of research data processing.

[0003] Existing distributed task scheduling technologies face numerous challenges when processing Mycobacterium tuberculosis research data. First, Mycobacterium tuberculosis research involves various types of computational tasks, including gene sequence alignment, protein structure prediction, drug resistance gene analysis, and molecular dynamics simulations. These tasks differ significantly in their computational characteristics; some are CPU-intensive, some memory-intensive, and some I / O-intensive. Traditional scheduling methods lack the ability to finely perceive task characteristics, making it difficult to allocate resources specifically based on task characteristics. Second, the processing flow of Mycobacterium tuberculosis research data typically involves complex dependencies. For example, genome assembly requires sequence quality control beforehand, and variant detection requires genome alignment beforehand. These complex dependencies further complicate task scheduling.

[0004] Chinese invention patent CN113296907A discloses a cluster-based task scheduling method. This method retrieves historical task data from the cluster and stores it in a database during a first time period. Then, it retrieves task data from the database for analysis during a second time period. Finally, it uses a task scheduling calculation model to optimize task scheduling parameters and update the cluster's task scheduling strategy. This method can determine indicators such as task resource usage, resource waste, data skewness, and execution time skewness based on historical data analysis, thereby optimizing the scheduling strategy. However, this method has the following shortcomings: First, it uses a fixed time period for data collection and analysis, making it impossible to perceive the dynamic changes in task characteristics in real time, and the response to sudden high-priority tasks in tuberculosis research is not timely enough; Second, it lacks the ability to model complex dependencies between tasks, and cannot model the workflow of tuberculosis research as a directed acyclic graph for systematic analysis; Third, it does not introduce a task characteristic perception mechanism, and cannot perform differentiated scheduling based on the CPU intensity, memory requirements, I / O characteristics, etc. of different types of tuberculosis research tasks; Fourth, it lacks an adaptive resource reservation mechanism, and cannot effectively predict the resource requirements of tasks and perform dynamic resource reservation, which easily leads to resource fragmentation and system idleness; Fifth, it does not use graph theory algorithms for task dependency analysis and critical path identification, making it difficult to achieve globally optimal scheduling decisions.

[0005] To address the shortcomings of the existing technologies, there is an urgent need for a distributed computing task scheduling optimization method that can sense task characteristics, analyze task dependencies, and implement adaptive resource reservation, so as to improve the efficiency and resource utilization of tuberculosis research data processing. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a distributed computing task scheduling optimization method and system for Mycobacterium tuberculosis research data, aiming to solve technical problems such as low task scheduling efficiency, low resource utilization, and improper handling of task dependencies in Mycobacterium tuberculosis research data processing.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A distributed computing task scheduling optimization method for Mycobacterium tuberculosis research data includes: acquiring a set of computing tasks for the Mycobacterium tuberculosis research data; extracting task feature vectors for each computing task in the set, including CPU intensity features, memory requirement features, I / O characteristic features, and data dependency features; constructing a task dependency directed acyclic graph (DAG) based on the computing task set, where nodes represent computing tasks and edges represent data dependencies between tasks; performing topological sorting on the task dependency DAG to determine the topological sequence of task execution; and identifying critical paths in the task dependency DAG based on a critical path analysis algorithm, where the critical path is the longest execution time path from the start node to the end node; and then optimizing the scheduling based on the distributed computing cluster. Based on the resource status information and task feature vectors of each computing node, a multidimensional fitness matrix for task-resource matching is constructed. The element values ​​of the multidimensional fitness matrix represent the degree of matching between computing tasks and computing nodes. Based on the critical path, topology sequence, and multidimensional fitness matrix, a scheduling strategy combining dynamic programming and heuristic search is adopted to determine the allocation scheme of computing tasks on each computing node. According to historical task execution statistics and current system load status, the resource requirements and expected execution time of tasks to be executed are predicted through an adaptive resource reservation mechanism, and the resource reservation amount is dynamically adjusted based on the resource pool management strategy. The allocation scheme is executed, and the computing tasks in the computing task set are distributed to the corresponding computing nodes for execution, while the task execution status and resource utilization are monitored in real time.

[0009] A distributed computing task scheduling optimization system for Mycobacterium tuberculosis research data includes: a task feature extraction module for acquiring a set of computing tasks for the Mycobacterium tuberculosis research data and extracting a task feature vector for each computing task in the set; a dependency graph construction module for constructing a directed acyclic graph (DAG) of task dependencies based on the set of computing tasks; a critical path analysis module for performing topological sorting on the DAG of task dependencies, determining the topological sequence of task execution, and identifying critical paths in the DAG of task dependencies based on a critical path analysis algorithm; and a fitness calculation module for constructing a task-to-feature graph based on the resource status information and task feature vectors of each computing node in the distributed computing cluster. The system comprises a multidimensional fitness matrix for resource matching; a scheduling optimization module, which uses a scheduling strategy combining dynamic programming and heuristic search based on the critical path, topology sequence, and multidimensional fitness matrix to determine the allocation scheme of computing tasks on each computing node; a resource reservation module, which predicts the resource requirements and expected execution time of tasks to be executed through an adaptive resource reservation mechanism based on historical task execution statistics and current system load status, and dynamically adjusts the resource reservation amount based on resource pool management strategies; and a task execution module, which executes the allocation scheme, distributes computing tasks in the computing task set to the corresponding computing nodes for execution, and monitors the task execution status and resource utilization in real time.

[0010] The beneficial effects of this invention include: First, by introducing a task feature perception mechanism, it can accurately identify key features of tuberculosis research tasks such as CPU intensity, memory requirements, and I / O characteristics, achieving precise matching between tasks and computing resources, improving resource utilization by approximately 25% to 35% compared to existing technologies; Second, by employing a graph theory-based task dependency analysis method, the complex tuberculosis research workflow is modeled as a directed acyclic graph, and the optimal execution sequence is determined through topological sorting and critical path analysis, effectively avoiding deadlocks and resource conflicts in task execution, reducing task waiting time by approximately 30% to 40% compared to existing technologies; Third, by combining dynamic programming and... The heuristic search-based hybrid scheduling strategy achieves a synergistic balance between critical path optimization and global load balancing, ensuring that scheduling decisions guarantee the timeliness of critical tasks while also considering the execution efficiency of non-critical tasks, exhibiting a non-linear efficiency enhancement characteristic of 1+1>2. Fourth, the adaptive resource reservation mechanism can intelligently predict resource demand based on historical data and current load status, effectively avoiding resource fragmentation through resource pooling management and preemptive scheduling strategies, reducing system idle rate by approximately 20% to 30% compared to existing technologies. Fifth, the real-time monitoring and closed-loop feedback mechanism ensures that the scheduling strategy can be continuously optimized to adapt to the dynamic changes in the data processing needs of tuberculosis research. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the distributed computing task scheduling optimization system for Mycobacterium tuberculosis research data of the present invention;

[0012] Figure 2 This is a flowchart illustrating the distributed computing task scheduling optimization method for Mycobacterium tuberculosis research data of the present invention.

[0013] Figure 3 This is a schematic diagram illustrating the calculation of the multidimensional fitness matrix of this invention;

[0014] Figure 4 This is a schematic diagram illustrating the collaborative scheduling of dynamic programming and heuristic search in this invention;

[0015] Figure 5 This is a schematic diagram of the adaptive resource reservation mechanism of the present invention. Detailed Implementation

[0016] Please refer to the attached document. Figures 1-5 The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0017] Reference Figure 1This invention provides a distributed computing task scheduling optimization system for Mycobacterium tuberculosis research data, including a task feature extraction module 1, a dependency graph construction module 2, a critical path analysis module 3, a fitness calculation module 4, a scheduling optimization module 5, a resource reservation module 6, and a task execution module 7.

[0018] The task feature extraction module 1 is used to acquire a set of computational tasks for Mycobacterium tuberculosis research data, and extracts a task feature vector for each computational task in the set. In one embodiment of the present invention, the computational tasks for Mycobacterium tuberculosis research data mainly include four categories: gene sequence alignment tasks, protein structure prediction tasks, drug resistance gene analysis tasks, and epidemiological association analysis tasks. For each computational task, the task feature extraction module 1 extracts a task feature vector containing four dimensions: CPU intensity features, memory requirement features, I / O characteristic features, and data dependency features.

[0019] For extracting CPU-intensive features, task feature extraction module 1 quantifies the computational complexity of the task and the expected number of floating-point operations. Taking a gene sequence alignment task as an example, assuming the length of the sequence to be aligned is... The time complexity coefficient of the comparison algorithm is The following innovative formula is used to calculate the CPU density feature value:

[0020] ,

[0021] in, This represents the CPU density feature value. To compare the time complexity coefficients of the algorithms, For sequence length, The sequence length influence factor. For CPU feature weight coefficients, It is the natural logarithm function. In a preferred embodiment, for the Smith-Waterman algorithm, The value is 1.2. The value is 2.0. The value is 1.0. For the BLAST algorithm, The value is 0.8. The value is set to 1.5. This formula avoids the computational overflow problem caused by excessively large values ​​through logarithmic transformation, while preserving the non-linear effect of sequence length on computational complexity.

[0022] For extracting memory requirement features, task feature extraction module 1 estimates the amount of data that needs to be loaded into memory during task processing. Taking protein structure prediction task as an example, assuming the protein has 100 amino acid residues... The spatial dimension of the structural model is The memory requirement characteristic value is calculated using the following innovative formula:

[0023] ,

[0024] in, This represents the memory requirement characteristic value, in MB. For memory coefficients, The number of amino acid residues. For the spatial dimension of the structural model, Based on the basic memory overhead factor, express The square root. In a preferred embodiment, The value is 0.05. The value is 3 (in three-dimensional space). The value is 10.0. The first term of this formula reflects the main memory usage of the protein structure data, and the second term reflects the auxiliary memory overhead during algorithm execution.

[0025] For I / O characteristic feature extraction, task feature extraction module 1 quantifies the data based on disk read / write frequency and data block size during task execution. Taking the drug resistance gene analysis task as an example, assuming the number of genome files to be read is... The average file size is The frequency of read and write operations is The following innovative formula is used to calculate the characteristic values ​​of I / O features:

[0026] ,

[0027] in, These are the characteristic values ​​of I / O features. For the number of genome files, Average file size, in GB. This refers to the read / write operation frequency, measured in times per second. Disk I / O bandwidth, measured in GB / s. This is the I / O efficiency coefficient. In a preferred embodiment, A value of 0.85 indicates that the actual I / O efficiency is 85% of the theoretical bandwidth. This formula accurately reflects the degree to which a task depends on disk I / O resources.

[0028] For data dependency feature extraction, task feature extraction module 1 analyzes the source of input data and the destination of output data for the task. Taking an epidemiological association analysis task as an example, assuming the number of prerequisite tasks depends on the task is... The number of subsequent tasks generated is The calculation of data-dependent feature values ​​uses the following formula:

[0029] ,

[0030] in, For data-dependent feature values, For the preceding dependency weight, This represents the number of prerequisite tasks. For subsequent dependency weights, This represents the number of subsequent tasks. In a preferred embodiment, The value is 1.5. A value of 1.0 is used to give higher weight to prerequisites, since the completion status of prerequisite tasks directly affects whether the current task can start.

[0031] Task feature extraction module 1 combines the feature values ​​from the above four dimensions into a task feature vector. This vector comprehensively characterizes the computational properties of the tuberculosis research task.

[0032] Dependency graph construction module 2 is used to construct a directed acyclic graph of task dependencies based on a set of computational tasks. Module 2 first parses the input data requirements and output data outputs of each computational task. Assume the set of computational tasks contains... One task, denoted as Dependency Graph Building Module 2 creates a directed acyclic graph. ,in For a set of nodes, For each task, there is a set of edges. Corresponding to a node in the diagram .

[0033] Dependency graph building module 2 traverses all task pairs, if task The output data is the task If the input data is sufficient, then add a slave node to the graph. Pointing to node Directed edge Meanwhile, dependency graph construction module 2 sets a weight value for each edge. , indicating from the task Transmit to task The amount of data. The formula for calculating edge weights is:

[0034] ,

[0035] in, The edge weight is expressed in MB. The amount of data transmitted, measured in MB. This represents the transmission overhead factor. In a preferred embodiment, The value is 1.1, which takes into account the protocol overhead during data transmission.

[0036] The dependency graph construction module 2 also needs to verify whether the constructed directed graph satisfies acyclicity. The verification method uses a depth-first search algorithm to traverse all nodes in the graph. If it is found that starting from a certain node, one can return to that node itself, it indicates that a cycle exists. For the detected cycle, the dependency graph construction module 2 decomposes a task in the cycle into multiple sub-tasks, thereby eliminating the cycle.

[0037] The critical path analysis module 3 performs topological sorting on the task-dependent directed acyclic graph (DAG), determines the topological sequence of task execution, and identifies the critical path in the DAG based on the critical path analysis algorithm. The critical path analysis module 3 first executes the topological sorting algorithm. The basic idea of ​​topological sorting is: starting with a node with an in-degree of 0, add it sequentially to the topological sequence, and delete all outgoing edges from that node. This process is repeated until all nodes have been added to the topological sequence.

[0038] Critical path analysis module 3 calculates the earliest start time (EST) and latest start time (LST) for each node. For each node... The formula for calculating its earliest start time is:

[0039] ,

[0040] in, For nodes The earliest start time, For nodes The set of all predecessor nodes, Predecessor node The earliest start time, For nodes Execution time, For the node To the node The data transmission time. For a starting node with an in-degree of 0, its earliest start time is 0.

[0041] Node execution time The estimation is based on the task feature vector and the processing power of the computing nodes. Assume the processing speed of the computing nodes is... ,but:

[0042] ,

[0043] in, For nodes Execution time, For nodes The CPU intensity characteristic value of the corresponding task. To improve the processing speed of computing nodes, The task initialization time. In a preferred embodiment, The value is 0.5 seconds.

[0044] Data transmission time The calculation formula is:

[0045] ,

[0046] in, For data transmission time, The edge weight (data volume). For network transmission bandwidth, This is due to network latency. In a preferred embodiment, The value is 0.01 seconds.

[0047] Critical path analysis module 3 uses a reverse traversal approach to calculate the latest start time of each node. For nodes... The formula for calculating its latest start time is:

[0048] ,

[0049] in, For nodes The latest start time, For nodes The set of all successor nodes, For successor nodes The latest start time, For data transmission time, For nodes The execution time. For a terminal node with an out-degree of 0, its latest start time is equal to its earliest start time.

[0050] The critical path analysis module 3 identifies critical nodes by comparing the earliest and latest start times of each node. If a node... satisfy If a node is not found in the list of critical nodes, then that node is a critical node. The path consisting of all critical nodes is the critical path. The total execution time of the critical path determines the lower bound of the shortest completion time for the entire task set.

[0051] The fitness calculation module 4 is used to construct a multi-dimensional fitness matrix for task-resource matching based on the resource status information and task feature vectors of each computing node in the distributed computing cluster. (Refer to...) Figure 3 Assuming the distributed computing cluster contains There are 1 computing node, denoted as _ ... The fitness calculation module 4 first obtains the resource status information of each computing node, including the number of available CPU cores. Available memory capacity Disk I / O bandwidth and network transmission bandwidth .

[0052] Fitness calculation module 4 constructs a Multidimensional fitness matrix ,in For the number of tasks, To calculate the number of nodes. Matrix elements. Indicates task With computing nodes The degree of matching between them. The calculation employs the following innovative multidimensional fitness evaluation formula:

[0053]

[0054] in, For the task With computing nodes fitness value, To calculate the matching degree in the efficiency dimension, For the matching degree of the memory matching dimension, Matching degree for I / O dimensions. For the matching degree of the communication overhead dimension, , , , These are the weight coefficients for each dimension, satisfying... In a preferred embodiment, The value is 0.35. The value is 0.30. The value is 0.20. The value is 0.15.

[0055] Matching degree of computational efficiency dimension The calculation formula is:

[0056] ,

[0057] in, To calculate the matching degree in the efficiency dimension, For computing nodes The number of available CPU cores, For the task CPU density characteristic value, To calculate the efficiency adjustment coefficient. In a preferred embodiment, The value is 1.2. When The larger the value, the better the computing power of the computing node can meet the computing requirements of the task.

[0058] Matching degree of memory matching dimension The calculation formula is:

[0059] ,

[0060] in, For the matching degree of the memory matching dimension, For computing nodes Available memory capacity For the task The memory requirement characteristics, express and The absolute value of the difference. This formula ensures that a positive match value is obtained only when the available memory of the compute node is greater than or equal to the memory requirement of the task.

[0061] I / O matching dimension matching degree The calculation formula is:

[0062] ,

[0063] in, Matching degree for I / O dimensions. For computing nodes Disk I / O bandwidth, For the task I / O characteristic values, This is a function that takes the minimum value. The formula limits the match to between 0 and 1.

[0064] Matching degree of communication overhead dimension Considering the data dependencies of the task and the network communication costs between nodes, the calculation formula is as follows:

[0065] ,

[0066] in, For the matching degree of the communication overhead dimension, For communication sensitivity coefficient, For the task Corresponding node The set of all predecessor nodes, To carry out the mission Assigned to compute nodes At that time, from the pioneer mission Communication time for data transmission It is an exponential function. In a preferred embodiment, The value is 0.1. This formula uses an exponential decay function, meaning the matching degree decreases as communication overhead increases.

[0067] The fitness calculation module 4 also introduces a synergy factor to evaluate the overall synergy of the task allocation scheme. When related tasks are assigned to adjacent computing nodes, data transmission latency can be reduced, resulting in a synergistic effect where 1+1>2. The formula for calculating the synergy factor is:

[0068] ,

[0069] in, As a synergistic effect factor, The task depends on the set of edges in the directed acyclic graph. This represents a directed edge in the graph. For the task and tasks Assigned compute nodes and Adjacency gain coefficients between This indicates a multiplication operation. If two tasks are assigned to the same computing node, the adjacency gain coefficient is 0.3; if they are assigned to adjacent computing nodes in the network topology, the adjacency gain coefficient is 0.15; if they are assigned to non-adjacent computing nodes in the network topology, the adjacency gain coefficient is 0.

[0070] The scheduling optimization module 5 is used to determine the allocation scheme of computing tasks on each computing node based on the critical path, topology sequence, and multidimensional fitness matrix, employing a scheduling strategy combining dynamic programming and heuristic search. (Refer to...) Figure 4 The core innovation of scheduling optimization module 5 lies in organically combining the global optimality of dynamic programming with the computational efficiency of heuristic search to form a mutually reinforcing collaborative optimization mechanism.

[0071] The scheduling optimization module 5 first uses a dynamic programming algorithm to prioritize tasks on the critical path. The state definition for dynamic programming is: This indicates that the critical path will be moved forward. After the first task is assigned, the second... Each task is assigned to a compute node. The minimum total execution time. The state transition equation is:

[0072] ,

[0073] in, For dynamic programming state values, This refers to the sequence number of the task on the critical path. For the number of the compute node, To calculate the total number of nodes, The compute node number assigned to the previous task. For the first critical path One task, To carry out the mission Assigned to compute nodes Execution time, For compute nodes To compute node Data transmission time, This is a function that takes the minimum value. Initial state. This represents the execution time of the first task on the critical path, which is assigned to each computing node.

[0074] The scheduling optimization module 5 determines the optimal allocation scheme for each task on the critical path by backtracking the dynamic programming table. This allocation scheme ensures that the execution time of the critical path is minimized, thereby guaranteeing the optimal lower bound of the overall scheduling time.

[0075] For tasks on non-critical paths, scheduling optimization module 5 uses a heuristic search algorithm for allocation. The heuristic function comprehensively considers the fitness values ​​of tasks and computing nodes, the current load status of computing nodes, and the impact on the execution of tasks on critical paths. The heuristic evaluation function is:

[0076] ,

[0077] in, This is a heuristic evaluation value. For the task With computing nodes fitness value, For computing nodes The current load, For computing nodes Total capacity Assign factors that affect the critical path to tasks. The calculation formula is:

[0078] ,

[0079] in, As the impact factor, To influence the sensitivity coefficient, To carry out the mission Assigned to compute nodes Potential delays in critical path tasks It is an exponential function. In a preferred embodiment, The value is 0.5.

[0080] The scheduling optimization module 5 processes non-critical path tasks sequentially according to the topology sequence, and allocates each task to the computing node with the highest heuristic evaluation value. During the allocation process, the scheduling optimization module 5 dynamically updates the load status of each computing node to ensure load balance.

[0081] The synergistic effect of dynamic programming and heuristic search is reflected in the following: dynamic programming determines the time-optimal allocation scheme for critical path tasks, while heuristic search, based on this, performs load balancing allocation for non-critical path tasks. The two mutually promote each other, ensuring both the execution efficiency of the critical path and the full utilization of computing resources. This synergistic mechanism produces a non-linear synergistic effect, making the overall scheduling performance superior to using either method alone.

[0082] Resource reservation module 6 is used to predict the resource requirements and expected execution time of tasks to be executed based on historical task execution statistics and current system load status through an adaptive resource reservation mechanism, and dynamically adjusts the resource reservation amount based on resource pool management strategies. (Refer to...) Figure 5 The resource reservation module 6 establishes a task resource demand prediction model. This model is based on historical execution statistics and uses an exponential smoothing prediction method.

[0083] For the task The resource demand forecast is given by the following formula:

[0084] ,

[0085] in, For the task At any moment The predicted resource demand, For smoothing coefficients, For the task At any moment The actual amount of resources used For the task At any moment The predicted resource demand. In a preferred embodiment, A value of 0.3 strikes a balance between responsiveness and stability.

[0086] Resource reservation module 6 dynamically adjusts the resource reservation coefficient based on the current system load. The formula for calculating the resource reservation coefficient is:

[0087] ,

[0088] in, To reserve a coefficient for resources, Basic reserve coefficient, For adjustment coefficients, This represents the current total system load. This represents the total system capacity. In a preferred embodiment, The value is 1.1. The value is set to 0.2. When the system load is high, the resource reservation coefficient is increased accordingly to cope with possible resource contention.

[0089] The actual amount of resources reserved is:

[0090] ,

[0091] in, The actual amount of resources reserved. To predict resource demand, Reserve a coefficient for resources.

[0092] Resource reservation module 6 employs a resource pooling management strategy, dividing the resources of the distributed computing cluster into multiple resource pools. These resource pools include dedicated resource pools and shared resource pools. Dedicated resource pools reserve fixed resources for high-priority tasks, while shared resource pools are used by ordinary tasks. Resources between resource pools can be dynamically allocated based on load conditions.

[0093] Resource reservation module 6 also implements a preemptive scheduling strategy. When a high-priority, sudden task arrives, if the resources in the shared resource pool are insufficient, it can preempt resources from a low-priority task that is currently executing. The preemption decision is based on an evaluation of task priority and resource scarcity. The formula for calculating the preemption cost is:

[0094] ,

[0095] in, To seize the price, The number of tasks that were taken over. For the task The time cost of saving checkpoints For the task The time overhead of resuming execution This indicates a summation operation. Preemption is only performed when the urgency of a higher-priority task exceeds the cost of preemption.

[0096] Resource pooling management effectively avoids the problem of resource fragmentation. Traditional resource allocation methods easily lead to resource fragmentation, meaning that although the overall system resources are sufficient, the available resources on a single computing node are insufficient to meet task requirements. Through resource pooling management, the resource reservation module 6 can aggregate resources across computing nodes, providing contiguous resource blocks for tasks.

[0097] The task execution module 7 executes the allocation scheme, distributing computational tasks from the computation task set to the corresponding computing nodes for execution, and monitoring the task execution status and resource utilization in real time. Based on the allocation scheme determined by the scheduling optimization module 5, the task execution module 7 sends tasks to the designated computing nodes. During task execution, the task execution module 7 continuously monitors the following metrics: actual task execution progress, CPU utilization of the computing nodes, memory utilization, disk I / O throughput, and network transmission bandwidth utilization.

[0098] Task execution module 7 calculates the deviation between the actual and expected progress of the task. The formula for calculating the deviation is:

[0099] ,

[0100] in, Schedule deviation rate The actual progress of the task execution. For the expected execution schedule of the task, express and The absolute value of the difference. If the progress deviation rate exceeds a preset threshold (set to 0.2 in the preferred embodiment), the task execution module 7 triggers a task rescheduling mechanism. The rescheduling mechanism includes: migrating the task to a less loaded computing node, adjusting the resource allocation of the task, or splitting the task into parallel components.

[0101] The task execution module 7 also records performance metrics data for task execution, including actual execution time, actual resource consumption, and data transfer volume. This data is used to update historical execution statistics and provide feedback for the resource reservation module 6 to predict resource demand.

[0102] After task scheduling is completed, task execution module 7 calculates the overall cluster resource utilization and task completion time. The formula for calculating the overall cluster resource utilization is:

[0103] ,

[0104] in, For the overall resource utilization of the cluster, To calculate the number of nodes, For computing nodes The load capacity, For computing nodes Total capacity This indicates a summation operation.

[0105] The task completion time is the total time for the entire task set to complete execution. Task execution module 7 evaluates the optimization effect of the scheduling scheme based on resource utilization and task completion time, and feeds the evaluation results back to the adaptive resource reservation mechanism, forming a closed-loop optimization. Through continuous feedback and adjustment, the system can continuously learn and adapt to changes in the data processing needs of Mycobacterium tuberculosis research, achieving continuous optimization of the scheduling strategy.

[0106] The modules of this invention have a close collaborative relationship: the task feature extraction module 1 provides task information to the dependency graph construction module 2; the dependency graph construction module 2 provides the graph structure to the critical path analysis module 3; the critical path analysis module 3 provides critical path information to the scheduling optimization module 5; the fitness calculation module 4 provides the fitness matrix to the scheduling optimization module 5; the resource reservation module 6 provides resource reservation information to the scheduling optimization module 5; and the task execution module 7 executes the allocation scheme of the scheduling optimization module 5 and provides feedback data to the resource reservation module 6. This collaborative effect between modules forms a closed-loop optimization system, achieving global optimization of task scheduling.

[0107] Reference Figure 2 This invention also provides a distributed computing task scheduling optimization method for Mycobacterium tuberculosis research data, the specific process of which is as follows:

[0108] Step 1: Obtain the set of computational tasks for Mycobacterium tuberculosis research data, and extract the task feature vector for each computational task in the set.

[0109] Step 2: Construct a task-dependent directed acyclic graph based on the set of computational tasks.

[0110] Step 3: Perform topological sorting on the task-dependent directed acyclic graph to determine the topological sequence of task execution, and identify the critical path in the task-dependent directed acyclic graph based on the critical path analysis algorithm.

[0111] Step 4: Based on the resource status information and task feature vectors of each computing node in the distributed computing cluster, construct a multi-dimensional fitness matrix for task-resource matching.

[0112] Step 5: Based on the critical path, topology sequence, and multidimensional fitness matrix, a scheduling strategy combining dynamic programming and heuristic search is adopted to determine the allocation scheme of computing tasks on each computing node.

[0113] Step 6: Based on historical task execution statistics and current system load status, predict the resource requirements and expected execution time of tasks to be executed through an adaptive resource reservation mechanism, and dynamically adjust the resource reservation amount based on the resource pool management strategy.

[0114] Step 7: Execute the allocation scheme, distribute the computing tasks in the computing task set to the corresponding computing nodes for execution, and monitor the task execution status and resource utilization in real time.

[0115] In one specific application embodiment of the present invention, a tuberculosis research institution needs to analyze the whole-genome sequencing data of 100 Mycobacterium tuberculosis strains. The analysis process includes multiple computational tasks such as sequence quality control, genome assembly, gene prediction and annotation, drug resistance gene detection, SNP variant analysis, and phylogenetic tree construction. These tasks have complex dependencies; for example, genome assembly depends on the completion of sequence quality control, and drug resistance gene detection depends on the completion of gene prediction and annotation.

[0116] Using the method of this invention, the task feature extraction module 1 first extracts feature vectors for each task. The sequence quality control task has high I / O characteristic feature values ​​because it requires reading a large amount of raw sequencing data; the genome assembly task has high CPU intensity and memory requirement feature values ​​because the assembly algorithm is computationally complex and requires a large amount of memory; the phylogenetic tree construction task has high data dependency feature values ​​because it depends on the output of all preceding tasks.

[0117] The dependency graph construction module 2 models these tasks and their dependencies as a directed acyclic graph. The critical path analysis module 3 identifies the critical path as: sequence quality control → genome assembly → gene prediction and annotation → SNP variation analysis → phylogenetic tree construction. This path has the longest execution time, determining the shortest completion time for the overall analysis process.

[0118] The fitness calculation module 4 constructs a multi-dimensional fitness matrix for task-resource matching based on the resource status of each computing node in the research institution's distributed computing cluster (including 8 computing nodes with configurations of 16-core CPU / 64GB memory, 32-core CPU / 128GB memory, 16-core CPU / 128GB memory, etc.).

[0119] The scheduling optimization module 5 uses a dynamic programming algorithm to determine the optimal allocation scheme for tasks on the critical path: sequence quality control tasks are allocated to nodes with high I / O bandwidth, genome assembly tasks are allocated to nodes with high CPU core counts and large memory, and SNP variant analysis tasks are allocated to nodes with high CPU core counts. For drug resistance gene detection tasks on non-critical paths, a heuristic search algorithm is used to allocate them to nodes with lighter loads to achieve load balancing.

[0120] Resource reservation module 6 predicts the resource requirements of each task based on historical data and reserves dedicated resources for high-priority urgent tasks. Task execution module 7 monitors the task execution status in real time. When it finds that the execution progress of the genome assembly task is lower than expected, it promptly triggers the rescheduling mechanism to migrate some assembly tasks to other computing nodes.

[0121] Using the method of this invention, the completion time of the whole genome analysis process of this tuberculosis research institution was shortened from 48 hours to 32 hours, and the resource utilization rate was increased from 65% to 88%, proving the effectiveness and superiority of the method of this invention.

[0122] In summary, this invention achieves comprehensive optimization of distributed computing task scheduling for Mycobacterium tuberculosis research data by organically combining innovative technologies such as task feature perception, graph theory dependency analysis, multidimensional fitness evaluation, dynamic programming and heuristic search collaborative scheduling, and adaptive resource reservation. The various technical modules form a close synergistic effect, jointly promoting the improvement of scheduling performance, exhibiting a non-linear efficiency enhancement characteristic of 1+1>2, and demonstrating significant progress and practicality compared to existing technologies.

Claims

1. A distributed computing task scheduling optimization method for Mycobacterium tuberculosis research data, characterized in that, include: A set of computational tasks for acquiring tuberculosis research data is provided. For each computational task in the set of computational tasks, a task feature vector is extracted. The task feature vector includes CPU intensity features, memory requirement features, I / O characteristic features, and data dependency features. A task-dependent directed acyclic graph is constructed based on the set of computing tasks, wherein the nodes of the task-dependent directed acyclic graph represent computing tasks, and the edges represent data dependencies between tasks. Perform topological sorting on the task-dependent directed acyclic graph to determine the topological sequence of task execution, and identify the critical path in the task-dependent directed acyclic graph based on the critical path analysis algorithm. The critical path is the longest execution time path from the start node to the end node. Based on the resource status information of each computing node in the distributed computing cluster and the task feature vector, a multidimensional fitness matrix for task-resource matching is constructed. The element values ​​of the multidimensional fitness matrix represent the degree of matching between the computing task and the computing node. Based on the critical path, the topology sequence, and the multidimensional fitness matrix, a scheduling strategy combining dynamic programming and heuristic search is adopted to determine the allocation scheme of computing tasks on each computing node. Based on historical task execution statistics and current system load status, the resource requirements and expected execution time of tasks to be executed are predicted through an adaptive resource reservation mechanism, and the resource reservation amount is dynamically adjusted based on the resource pool management strategy. The allocation scheme is executed to distribute the computing tasks in the computing task set to the corresponding computing nodes for execution, and the task execution status and resource utilization are monitored in real time.

2. The distributed computing task scheduling optimization method for Mycobacterium tuberculosis research data according to claim 1, characterized in that, The extracted task feature vector includes: For the task of aligning Mycobacterium tuberculosis gene sequences, the CPU density feature is determined based on sequence length and alignment algorithm complexity; For the task of predicting the protein structure of Mycobacterium tuberculosis, the memory requirement features are determined based on the protein molecular weight and structural model complexity. For the task of analyzing tuberculosis drug resistance, the I / O characteristics are determined based on the frequency of genomic data read / write and the size of data blocks; For the task of epidemiological association analysis of Mycobacterium tuberculosis, the data dependency features are determined based on the degree of spatiotemporal association between samples.

3. The distributed computing task scheduling optimization method for Mycobacterium tuberculosis research data according to claim 1, characterized in that, The construction task depends on a directed acyclic graph including: The input and output data streams between computation tasks are analyzed. If the output data of the first computation task is the input data of the second computation task, then a directed edge from the first computation task node to the second computation task node is added to the task-dependent directed acyclic graph. A weight value is set for the directed edge, and the weight value represents the amount of data transmitted. Verify the acyclicity of the directed acyclic graph that the task depends on; if a cycle is detected, perform task decomposition.

4. The distributed computing task scheduling optimization method for Mycobacterium tuberculosis research data according to claim 1, characterized in that, The critical path identification algorithm based on critical path analysis includes: Calculate the earliest start time and latest start time of each node in the directed acyclic graph that the task depends on; The node whose earliest start time is equal to its latest start time is identified as the critical node; The path formed by the key nodes is the key path. The lower bound of the overall scheduling time is determined based on the total execution time of the critical path.

5. The distributed computing task scheduling optimization method for Mycobacterium tuberculosis research data according to claim 1, characterized in that, The multidimensional fitness matrix for construction task-resource matching includes: Obtain the number of available CPU cores, available memory capacity, disk I / O bandwidth, and network transmission bandwidth for each computing node; For each combination of computing task and each computing node, a multidimensional fitness value is calculated based on the degree of matching between the task feature vector and the node resource status. The multidimensional fitness value comprehensively considers computational efficiency, memory matching, I / O matching, and communication overhead dimensions.

6. The distributed computing task scheduling optimization method for Mycobacterium tuberculosis research data according to claim 1, characterized in that, The scheduling strategy that combines dynamic programming and heuristic search includes: Dynamic programming algorithm is used to solve the optimal task allocation scheme on the critical path to ensure that the execution time of the critical path is minimized; A heuristic search algorithm is used to allocate non-critical path tasks, and the heuristic search is evaluated based on load balancing and resource utilization. Global scheduling optimization is achieved through the synergistic effect of dynamic programming and heuristic search.

7. The distributed computing task scheduling optimization method for Mycobacterium tuberculosis research data according to claim 1, characterized in that, The adaptive resource reservation mechanism includes: Establish a task resource demand prediction model based on historical execution statistics; The resource reservation coefficient is dynamically adjusted based on the current system load status. A preemptive scheduling strategy is used to handle high-priority, sudden task requests; Resource pooling management helps avoid resource fragmentation.

8. The distributed computing task scheduling optimization method for Mycobacterium tuberculosis research data according to claim 1, characterized in that, The real-time monitoring task execution status includes: Monitor the deviation between the actual execution progress and the expected progress of each computing task; If the deviation exceeds the preset threshold, the task rescheduling mechanism will be triggered; Recording task execution performance metrics data is used to update historical execution statistics.

9. The distributed computing task scheduling optimization method for Mycobacterium tuberculosis research data according to claim 1, characterized in that, Also includes: After task scheduling is completed, calculate the overall cluster resource utilization and task completion time; The optimization effect of the scheduling scheme is evaluated based on the resource utilization rate and task completion time. The evaluation results are fed back to the adaptive resource reservation mechanism to form a closed-loop optimization.

10. A distributed computing task scheduling and optimization system for Mycobacterium tuberculosis research data, used to execute the distributed computing task scheduling and optimization method for Mycobacterium tuberculosis research data as described in any one of claims 1-9, characterized in that, include: The task feature extraction module is used to acquire a set of computational tasks for Mycobacterium tuberculosis research data. For each computational task in the set of computational tasks, a task feature vector is extracted. The task feature vector includes CPU intensity features, memory requirement features, I / O characteristic features, and data dependency features. The dependency graph construction module is used to construct a task dependency directed acyclic graph based on the set of computing tasks, wherein the nodes of the task dependency directed acyclic graph represent computing tasks, and the edges represent data dependencies between tasks; The critical path analysis module is used to perform topological sorting on the task-dependent directed acyclic graph, determine the topological sequence of task execution, and identify the critical path in the task-dependent directed acyclic graph based on the critical path analysis algorithm. The fitness calculation module is used to construct a multi-dimensional fitness matrix for task-resource matching based on the resource status information of each computing node in the distributed computing cluster and the task feature vector. The scheduling optimization module is used to determine the allocation scheme of computing tasks on each computing node based on the critical path, the topology sequence and the multidimensional fitness matrix, using a scheduling strategy that combines dynamic programming and heuristic search. The resource reservation module is used to predict the resource requirements and expected execution time of tasks to be executed based on historical task execution statistics and current system load status through an adaptive resource reservation mechanism, and dynamically adjust the resource reservation amount based on the resource pool management strategy. The task execution module is used to execute the allocation scheme, distribute the computing tasks in the computing task set to the corresponding computing nodes for execution, and monitor the task execution status and resource utilization in real time.

Images