Sample batch processing scheduling system based on workflow engine

Abstract

The invention relates to a sample batch processing scheduling system based on a workflow engine, and belongs to the technical field of bioinformatics analysis. The system comprises a tool construction module used for configuring tool parameters, resource demand parameters and input and output specifications, generating command lines and constructing a running environment; the flow arrangement module is used for defining flow contents and establishing a dependency relationship between tools; the dependency checking module is used for constructing a directed acyclic graph and executing multi-stage legality detection; the process execution module is used for scheduling tool nodes according to the node dependency relationship and executing a process task; the process control module is used for managing and controlling the running process task; and the log management module is used for creating log files named by task numbers and generating independent logs for each tool node. According to the method, two operation modes of web page end and manual JSON configuration are supported, the usability and the flexibility are both considered, and the working efficiency can be improved.

Description

Technical Field

[0001] This invention belongs to the field of bioinformatics analysis technology and relates to a sample batch processing scheduling system based on a workflow engine. Background Technology

[0002] In the field of bioinformatics, data analysis heavily relies on complex specialized software tools and the workflows formed by their combinations. Current technical solutions primarily employ three implementation paths: First, deployment via command-line interface (CLI) or scripted methods (such as CellRanger parameter configuration and the Snakemake workflow engine). While these methods offer flexibility and scalability, they require users to possess programming skills, creating a significant technical barrier for non-specialist users (such as experimental researchers). Second, graphical workflow systems (such as Galaxy) reduce operational difficulty through drag-and-drop operations, but their preset components and fixed patterns limit the ability to deeply customize workflows. Third, general-purpose task scheduling platforms (such as Apache Airflow), while having built-in DAG validity checks, are not optimized for bioinformatics scenarios and struggle to handle dynamic data dependencies and tool heterogeneity. All of these technical paths have significant limitations—CLI and scripted deployments lack standardized description formats, leading to inefficient parameter management; graphical systems sacrifice flexibility for ease of use; and general-purpose scheduling platforms suffer from insufficient scenario adaptability, impacting practical application effectiveness.

[0003] Furthermore, existing technologies suffer from multiple contradictions in tool configuration and process management. On the one hand, non-professional users struggle to master command lines or scripts, while professionals lack a unified machine-readable description format (such as JSON) when deploying tools in batches, leading to cumbersome and error-prone parameter configuration. On the other hand, existing process orchestration methods fail to balance ease of use and flexibility: graphical systems simplify operations through fixed templates but limit the customization capabilities of complex processes; code-driven systems (such as Snakemake) support deep customization but require users to have programming skills, creating a technical barrier. More importantly, most solutions lack an automated DAG validity verification mechanism, and cyclic dependency errors are often only discovered at runtime, significantly increasing debugging costs. In addition, task management functions are scattered across different systems, lacking both a unified interface for intuitive monitoring by non-professional users and centralized management for fine-grained control by professionals (such as pausing and restarting tasks), resulting in insufficient process reliability and management efficiency.

[0004] Therefore, current bioinformatics research urgently needs to overcome existing technological bottlenecks. To address the problem of inefficient tool configuration, standardized description formats need to be established to improve deployment efficiency; to resolve the imbalance between ease of use and flexibility in workflow orchestration, a hybrid architecture that balances graphical operation and code-level control needs to be designed; regarding workflow reliability, an automated DAG verification mechanism should be introduced to avoid logical errors at the source; simultaneously, an integrated task management platform needs to be built to uniformly handle status monitoring, anomaly intervention, and other operations. These improvements not only require the integration of graphical usability, script-level flexibility, and intelligent verification mechanisms, but also need to meet the needs of large-scale tool deployment, complex workflow design, and multi-scenario adaptation through modular design and dynamic dependency handling capabilities, thereby bridging the technical gap between non-professional users and professionals. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a sample batch processing scheduling system based on a workflow engine. By integrating graphical ease of use, script-level flexibility and intelligent verification mechanism, and through modular design and dynamic dependency processing capabilities, it meets the needs of large-scale tool deployment, complex process design and multi-scenario adaptation, bridging the technical gap between non-professional users and professionals.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A sample batch processing scheduling system based on a workflow engine, comprising: The tool building module is used to configure tool description parameters, resource requirement parameters, input and output specifications, generate command lines, and build the runtime environment. The process orchestration module is used to define process content, and at the same time, it transforms the tools built by the tool building module into nodes of a directed acyclic graph and establishes the dependencies between tools. The dependency verification module constructs a directed acyclic graph and performs multi-stage legality checks to ensure that the process scheduling logic is acyclic and executable. The process execution module schedules tool nodes based on the node dependencies defined in the directed acyclic graph to execute process tasks; The process control module is used to manage and control running process tasks, including pausing, terminating, resuming, or restarting tasks. The log management module creates log files named after task numbers, appending records of status changes, key operations, and timestamps throughout the process; it also generates independent logs for each tool node during workflow execution.

[0008] Furthermore, the tool building module includes a tool metadata definition unit, a resource configuration unit, an IO specification definition unit, a command-line execution unit, and an environment building unit; The tool metadata definition unit is used to configure tool parameters, which include tool name, tool version, tool category, and tool description. The resource configuration unit is used to configure the resource requirements parameters of the tool. The resource requirements parameters include quantitative parameters such as the number of CPU cores, memory capacity, number of GPUs, and maximum runtime. The IO specification definition unit is used to set the input and output specifications, defining the input file paths required by the tool and the expected output file paths; The command-line execution unit is used to generate executable shell commands; The environment building block is used to build the environment in which the tool runs.

[0009] Furthermore, the command-line execution unit replaces all placeholders in the template with actual values ​​to generate a complete shell executable script.

[0010] Furthermore, the process orchestration module includes a process definition unit, a tool node generation unit, and a parameter dependency definition unit; The process definition unit is used to set the process name, version number, process description field and process category; the process name is used to identify the unique logical name of the process, the version number is used to track process iteration, the process description field is used to describe the purpose, inputs and outputs and applicable scenarios of the process, and the process category is used to identify the application area or analysis type of the process. The tool node generation unit instantiates the tools built by the tool building module into nodes of a directed acyclic graph; The parameter dependency definition unit establishes the dependency relationship between tools through the parameter passing rules between parent and child nodes.

[0011] Furthermore, instantiating tools into nodes of a directed acyclic graph includes identifying and parsing all tools selected by the user through the tool building module; and instantiating each tool into a node with specific attributes based on the tool parameters. Each node contains all the information required to execute the tool, including command-line scripts, environment variable settings, and resource requirements. The parameter passing rules include: the parent node is the tool node that generates the output result after execution, and the child node is the subsequent tool node that references the output result of the parent node through predefined mapping rules for the input parameters.

[0012] Furthermore, the dependency verification module constructs a directed acyclic graph and performs multi-stage validity checks to ensure that the process scheduling logic is acyclic and executable, including: Self-loop check: If a node has an edge pointing to itself, an error is immediately reported; Cycle dependency detection: Using depth-first search, by maintaining the set of visited nodes and the current recursive path stack, it detects whether there is a closed path. If a closed path is found, the specific cycle path is returned. Sub-process validation: The user specifies a start node and an end node. All intermediate nodes reachable from the start node and reachable from the end node are extracted, forming a logically closed subgraph. Then, forward and reverse reachability validation is performed, including traversing all reachable nodes along the dependency direction from the start node, denoted as the ForwardSet; and traversing all predecessor nodes along the reverse dependency direction from the end node, denoted as the BackwardSet. The subgraph node set is then the intersection of the ForwardSet and BackwardSet. Finally, the existence of isolated nodes is checked. Isolated nodes are nodes that exist between the start and end nodes but are not in the subgraph node set. Verification Results and Execution Guarantee: If verification fails, a structured error message is returned; if verification passes, a valid directed acyclic graph is output as the scheduling basis for the process execution module.

[0013] Furthermore, the process execution module schedules tool nodes based on node dependencies to execute process tasks, including: Load process metadata and all node configurations, and skip user-marked skip nodes; Create a dedicated working directory for each node that is not skipped and pre-generate a shell script; Check the dependency satisfaction of the node. If the dependency is satisfied, all input parameters of the node have obtained valid values ​​and all parent nodes have completed successfully. Before any node is executed, the system parses all input parameters of that node in real time and replaces the placeholders in the pre-generated Shell script with the actual values ​​corresponding to the input parameters. Submit the replaced Shell script and configure the scheduler type; Poll the scheduler status and read the rc file in the node's working directory to determine the execution result; the rc file contains the command exit code. If the command exit code is 0, the task execution is successful; otherwise, the task execution fails. If the task execution fails, immediately trigger a global failure, terminate all running nodes, and record the error log. For nodes that execute successfully, their output parameters are registered in the global context; subsequent nodes automatically retrieve the latest values ​​before triggering. After a process task is successfully executed, the output is a collection of business result files generated by the final node, while the task metadata and complete operation logs are persisted. When a process task fails to execute, the error information, rc value and context snapshot of the first failed node are returned for user diagnosis.

[0014] Furthermore, the process control module is used to manage and control the running process tasks; When pausing or terminating a task, first read the task metadata to identify the context of the current process task execution; then, depending on the scheduler type, call the job cancellation command or send an interrupt or forced termination signal to the operating system process to stop computation and release computing resources, while updating the global state of the task. When resuming a task, retain the completed results, clear the pause marker, and then continue execution from the last unfinished node; When rerunning a task, a new task instance with the Rerun tag is created, the references to the original task are preserved, the status flags of all nodes are cleared, and the process task is executed from the beginning.

[0015] The beneficial effects of this invention are as follows: This invention provides a sample batch processing scheduling system based on a workflow engine. This system supports both web-based and manual JSON configuration modes, balancing ease of use and flexibility. Non-professional users can easily configure tools and processes through a graphical interface and automatically generate standardized description files. Visual drag-and-drop functionality on the web interface lowers the barrier to entry for bioinformatics analysis for non-professional users. Professional users, on the other hand, can achieve fine-grained control over large-scale tool deployments and complex process structures by manually editing or batch importing description files of the same format, meeting the customization needs of advanced users.

[0016] Meanwhile, this invention incorporates an automated dependency verification module to perform directed acyclic graph (DAG) validation on the dependencies between tools before process execution, preventing execution failures caused by logical cycles and ensuring the reliability and schedulability of the process. Furthermore, this invention establishes a unified task management mechanism to achieve centralized dynamic control of running processes (including pausing, terminating, resuming, and restarting), and supports full lifecycle logging, viewing, retrieval, and statistical analysis, meeting needs from ease of operation to refined maintenance.

[0017] In summary, this invention balances ease of use and flexibility, helping researchers quickly build analysis workflows and adapting to diverse scenarios through modular integration and parameterized replacement, ultimately promoting the efficient reuse of bioinformatics tools and improving the efficiency of scientific research collaboration.

[0018] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 A block diagram of the task execution subsystem; Figure 2 Here is a block diagram of the task management subsystem. Figure 3 An example of configuring the tool in manual JSON mode; Figure 4 Example of configuring tools for web applications; Figure 5 An example of setting up parameter passing in manual JSON mode; Figure 6 Example of setting up parameter passing for a web page; Figure 7 This is a diagram illustrating dependency verification. Figure 8 This is a diagram illustrating the process execution records and results in JSON mode. Figure 9 This is a schematic diagram of the process execution record and execution results shown on a web page. Figure 10 This is a diagram illustrating flow control operations and log management in JSON mode. Figure 11 This is a schematic diagram of the process control operations and log management records shown on the web page. Detailed Implementation

[0020] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0021] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0022] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0023] The sample batch processing scheduling system based on a workflow engine provided by this invention includes two subsystems: a task execution subsystem and a task management subsystem.

[0024] The task execution subsystem performs the following operations: 1) Tool building: Supports two configuration modes.

[0025] First, the parameters of bioinformatics software (such as BWA, GATK, etc.) and server resource parameters can be configured through the web front-end interface, and tool parameter files (JSON format) can be automatically generated, making it easy for non-bioinformatics professionals to quickly build analysis tools as needed; Secondly, based on the predefined tool JSON format, it allows for manual batch configuration of bioinformatics software and server resource parameters, which is suitable for large-scale tool deployment.

[0026] 2) Workflow orchestration: Combines multiple tools into a complete analysis workflow, also supporting two modes.

[0027] First, by using a web-based visual drag-and-drop interface and filling in dependencies between tools, a process description file (JSON format) is automatically generated, lowering the barrier to entry for non-professional users. Secondly, based on the process JSON specification, the execution dependencies between tools are manually defined to satisfy advanced users' fine control over the process structure.

[0028] 3) Dependency verification: Construct a directed acyclic graph (DAG) based on the dependencies between tools, and perform a validity check on the process structure to ensure that the scheduling logic is acyclic and executable.

[0029] 4) Process execution: Start batch processing tasks based on the verified DAG, and schedule the tools to run in the order of dependency.

[0030] The task management subsystem performs the following operations: 1) Process control: Dynamically manage the running process, including pause, termination, resumption and restart operations.

[0031] 2) Log Management: Automatically records the operation logs of the entire process lifecycle and provides task viewing, retrieval and statistical analysis functions.

[0032] The specific content of the method of the present invention will be illustrated by an embodiment below.

[0033] Example 1 To better demonstrate the application of this invention in bioinformatics analysis, a transcriptome analysis example based on FastQC (quality control detection), Fastp (quality control), hisat2 (alignment to a reference genome), bam2count (quantitative gene expression), and DESeq2 (differential gene analysis) will be described below. This example illustrates the complete workflow from raw sequencing data processing to quantitative gene expression analysis.

[0034] 1. Tool building module like Figure 3 and Figure 4 As shown, the tool building module is used to configure the description parameters, resource requirements, input / output (IO) specifications, command line execution, and build environment of a bioinformatics analysis tool (taking Fastp as an example). It includes a tool metadata definition unit, a resource configuration unit, an IO specification definition unit, a command line execution unit, and an environment building unit, wherein: The tool metadata definition unit is used to configure tool parameters. In this embodiment, the tool description parameters include: tool name (multi_fastp), tool version (v0.01), tool category (transcriptomics), and tool description content (used for Fastp quality control in multi-omics sequencing samples).

[0035] The resource configuration unit is used to configure resource requirement parameters. These parameters are quantifiable indicators, including the number of CPU cores (e.g., 4), memory capacity (e.g., 16GB), number of GPUs (Fastp does not depend on GPUs, so set it to 0), and maximum runtime (1-00:00:00, i.e., 1 day). These parameters will be used as variables in subsequent command generation and task scheduling. For example, the number of CPU cores can be mapped to the thread parameter in the command line (-m 4).

[0036] The IO specification definition unit is used to define the input / output (IO) specification. This unit explicitly declares the input file paths required by the tool and the expected output file paths through variable naming rules.

[0037] When building tools, users specify semantic names for the aforementioned variables through the front-end interface or JSON configuration file. The system then uses this to establish an input / output contract based on a dynamic binding mechanism. The system supports two syntaxes for variable binding: one is the template syntax `${var}`, where placeholders such as `${sampleInfo}`, `${url}`, and `${cpus}` are directly used in command templates. During task execution, the system searches for the corresponding field values ​​in the current sample's metadata context and replaces them—for example, `${sampleInfo}` is replaced with " / mnt / ssddata / public / platform / datasets / test / transcriptome / samples_paired.csv", and `${cpus}` is replaced with 4. The other is the object property syntax `io.var`, used to explicitly declare references to IO specification variables in advanced configurations, such as `"input": "io.sampleInfo"`, which the system interprets as a binding to the `sampleInfo` variable. Both methods point to the same global variable pool, ensuring consistent access to input / output paths for components such as commands, scripts, and logs.

[0038] The command-line execution unit ultimately generates executable shell commands: This unit calls the ${batch_runner} batch executor according to the ${mode} branch (which takes the value "paired" or "single"), loads the sample information table via -i ${sampleInfo}-d ${column}, and embeds Fastp bioinformatics analysis commands in -cmd; where {i:${url}:0} and {i:${url}:1} dynamically extract the R1 / R2 paths of each row in the CSV, {d / (b:${url}:0)} and other rules generate the corresponding output paths, -h and -j generate HTML / JSON quality control reports, -m ${cpus} injects the number of threads, and -u "${outdir} / ${update_tab}" outputs the updated sample table for downstream tools to use. During the task instantiation phase, the system iterates through all sample information in the sample information table associated with sampleInfo and replaces all placeholders in the command template (including ${sampleInfo}, ${url}, ${mode}, ${outdir}, ${update_tab} from the IO specification definition unit and ${cpus} from the resource configuration unit) with the actual parameter values ​​entered by the user according to their personalized needs. Finally, a complete Shell script that can be directly submitted and executed is generated, realizing an efficient, automated, and traceable transcriptome data quality control tool.

[0039] The environment building unit is used to build the environment in which the tool runs. It is compatible with Conda virtual environment and Docker, Podman, and Apptainer container image technologies, enabling flexible configuration of the tool's runtime environment.

[0040] 2. Process orchestration module like Figure 5 and Figure 6 As shown, the process orchestration module includes a process definition unit, a tool node generation unit, and a parameter dependency definition unit, wherein: The workflow definition unit is used to define the workflow name (a unique logical name identifying the workflow), version number (used to track workflow iterations), workflow description field (briefly describing the workflow's purpose, inputs, outputs, and applicable scenarios), and workflow category (identifying the workflow's application area or analysis type). Taking the actual content of the transcriptomics workflow RNASeq1 as an example: the workflow name is RNASeq1, the version information is v1.0, the workflow description field is "used for standard RNA-seq data analysis," and the workflow category is transcriptomics.

[0041] The tool node generation unit is used to instantiate tools obtained through the tool building module into nodes of a directed acyclic graph (DAG), as follows: (1) Tool identification and parsing: First, the system needs to identify and parse all the tools selected by the user through the tool building module. Each tool has its unique input, output and configuration parameters.

[0042] (2) Tool instantiation: Based on the tool's metadata information (such as name, version, input / output specifications, etc.), the system instantiates each tool into a node with specific attributes. Each node contains all the information required to execute the tool, including but not limited to command-line scripts, environment variable settings, and resource requirements (CPU, memory, etc.).

[0043] (3) Based on the tool execution order specified by the user during the process design phase, the system determines the dependencies between tools and constructs a DAG accordingly. In this process, the system checks whether the output of the previous tool meets the requirement of depending on the input of the next tool, thereby establishing the logical relationship between parent and child nodes.

[0044] (4) The final DAG not only reflects the execution order between tools, but also supports the display to users in a graphical interface, helping users to intuitively understand the structure and data flow of the entire workflow.

[0045] The primary responsibility of the parameter dependency definition unit is to establish dependencies between tools based on the parameter passing rules between parent and child nodes. Specifically, the relationship between parent and child nodes is determined by the explicit parameter passing rules in the process: a parent node is a tool node that generates specific output results after its execution, while a child node is a subsequent tool node whose input parameters reference the output of its parent node through predefined mapping rules. The system automatically establishes parent-child dependencies based on the structured mapping relationships declared in the `param_passing` field—where each entry specifies the data flow through `source` (parent node tool name) and `target` (child node tool name), and precisely declares in `mappings` which output parameter of the parent node (e.g., `io.update_tab`) should be passed to which input parameter of the child node (e.g., `io.sampleInfo`). Taking Fastp → HISAT2 → bam2count in RNASeq1 analysis as an example, the actual `param_passing` configuration is as follows: Figure 5 As shown, this configuration indicates that the `sampleInfo` input of `multi_hisat2` receives the `update_tab` output of `multi_fastp` (i.e., the cleaned sample information table), and the `indir` input of `bam2counts` receives the `outdir` output of `multi_hisat2` (i.e., the alignment result directory). It is worth noting that this process does not use the "input parameter inheritance" mode (i.e., it does not rely on automatic variable name matching); all data transfer is achieved through the "output parameter passing" explicitly declared in `param_passing`, ensuring loose coupling between tools, clear interfaces, and precise process orchestration.

[0046] 3. Dependency Validation Module This module ensures that the process scheduling logic is acyclic and executable by constructing a directed acyclic graph (DAG) and performing multi-stage validity checks.

[0047] The system first generates a DAG represented by an adjacency list based on the dependencies between tool nodes (explicitly defined by `param_passing` or user connections). Then, a two-phase cycle detection is performed. (1) DAG Construction and Cycle Detection: Self-loop check: If a node has an edge pointing to itself (e.g., A → A), immediately report the error "Node X has a self-loop"; Cyclic dependency detection: Use depth-first search (DFS) to detect whether there is a closed path (e.g., A → B → C → A) by maintaining the "set of visited nodes" and the "current recursive path stack". Once found, return the specific cycle path "Cyclic dependency path: A→B→C→A".

[0048] (2) Sub-workflow support and subgraph verification: To support local debugging, incremental operation, or modular reuse, the system allows users to specify sub-workflows—that is, a continuous execution segment extracted from the complete workflow. Sub-workflow definition: Users explicitly specify a start node and an end node through the interface or command line. The system dynamically extracts all intermediate nodes that are reachable from the start node and can reach the end node, forming a logically closed subgraph. Taking RNASeq1 as an example, the HISAT2 → bam2count segment can be debugged, skipping the quality control steps before Fastp and reusing existing alignment results, only rerunning the bam2count → DESeq2 quantitative and differential analysis. To ensure connectivity within sub-processes, the system performs bidirectional reachability verification: Forward traversal: Starting from the starting node, traverse all reachable nodes along the dependency direction (parent → child), denoted as the ForwardSet set; Reverse traversal: Starting from the ending node, traverse all predecessor nodes in the reverse dependency direction (child ← parent), denoted as the BackwardSet set; Subgraph node = ForwardSet ∩ BackwardSet.

[0049] (3) Definition of an isolated node: In the context of a sub-process, an isolated node is a node that exists between the user-selected start and end points but is not in the ForwardSet ∩ BackwardSet. In other words, if a node cannot be reached from the start node or the end node, it is considered isolated and will be excluded from the sub-process execution plan. For example: The process is A → B → C → D → E, and the user specifies start = C and end = E. If there is a bypass node F (such as B → F), then F is not in the forward path of C, nor in the reverse path of E, so it is an isolated node and will not be included in the sub-process. If the subgraph is empty (such as the start node and the end node have no path connection) or contains a cycle, the validation will fail.

[0050] (4) Verification results and execution guarantee: Verification failure: Return structured error information, such as "node multi_fastp has a self-loop" or "there is an isolated node bam2count in the subprocess"; Verification pass: Output a valid DAG (or subgraph) as the scheduling basis for the process execution module, ensuring that the task starts strictly in the topological order, avoiding deadlock, resource competition or data not ready errors, thereby ensuring the reliability and executability of the process.

[0051] 4. Process Execution Module The process execution module dynamically schedules tool nodes based on the node dependencies defined in the DAG (including explicit parameter passing dependencies and user-specified sequence dependencies) to ensure correct data flow, reasonable resource allocation, and immediately terminates the entire process if any node fails.

[0052] The execution process is illustrated using the standard RNA-seq analysis workflow (RNASeq1) as an example: This workflow includes five core tool nodes: multi_fastqc → multi_fastp → multi_hisat2 → bam2counts → multi_deseq2_anno. Their dependencies are explicitly defined by param_passing (e.g., multi_fastp.update_tab → multi_hisat2.sampleInfo). The execution flow is as follows: (1) Initialization and skipped node processing: The system first loads the process metadata and all node configurations; "skipped nodes" are explicitly marked by the user through the UI or command line before the process starts (e.g., setting "skip": true), and are often used for debugging or reusing intermediate results. Skipped nodes do not generate scripts or allocate resources, and their output parameters cannot be referenced downstream.

[0053] (2) Script and directory pre-generation: Create a dedicated working directory for each node that is not skipped (named by the tool name and version); pre-generate an executable Shell script (such as tool_command.sh) based on the cmds template in the tool definition.

[0054] (3) Dependency satisfaction check and dynamic triggering: "Dependency satisfaction" means that all input parameters of a node have obtained valid values ​​(from user input, parent node output or system injection), and all parent nodes have completed successfully; the system continuously checks the status of each node through centralized polling (period 50 ms); (4) Input parameter parsing and path conversion: Before a node is executed, the system parses all input parameters of that node in real time. For example, the sampleInfo parameter value of multi_hisat2 may be a relative path ".. / fastp / update_samples.tab"; the relative path is automatically converted to an absolute path based on the current node's working directory (such as " / workspace / task_123 / node_multi_fastp / update_samples.tab"); the updated content is the placeholder in the pre-generated Shell script (such as ${sampleInfo}), which is replaced with the actual absolute path to ensure that the command can access the file correctly.

[0055] (5) Job submission and scheduler configuration: The submission content is a complete Shell script (including environment activation, command sequence, exit code capture, etc.); the scheduler type (SLURM / PBS / local) is usually specified in advance in the global process configuration or platform default settings; if not configured, the local mode is used by default.

[0056] (6) Execution monitoring and result determination: The system determines the execution result by polling the scheduler status (such as queue) and reading the rc file in the node directory; the rc file is automatically generated by the end command code in the tool shell script, and the content is the final exit code of the command (such as echo $?>rc), which contains only one line of integers (0 indicates success, non-zero indicates failure); if the rc value is non-zero, the system immediately triggers a global failure: terminates all running nodes and records the error log (including standard output / error, resource usage).

[0057] (7) Output propagation and data consistency: Successful nodes register their output parameters (such as io.count = " / workspace / ... / matrix_count.tab") to the "global context" shared during the process run. This is a key-value store that can be accessed by all nodes in the entire workflow and is used to dynamically pass intermediate results. Downstream nodes automatically read the latest valid value of the corresponding parameter from the "global context" before triggering, ensuring that the path in the script always points to the real output file and ensuring the consistency and reliability of data flow.

[0058] (8) Final execution result: When successful: The process output is a collection of business result files generated by the final node (such as the differentially expressed gene table deseq2_results.tsv and annotation files generated by multi_deseq2_anno), and the task metadata (including the input / output snapshots of each node, resource consumption, and execution time) and complete operation logs are persisted at the same time; When unsuccessful: The error information, rc value and context snapshot of the first failed node are returned for the user to diagnose.

[0059] The aforementioned tool building module, process orchestration module, dependency verification module, and process execution module are all components of the task execution subsystem.

[0060] 5. Process Control Module The process control module is used to perform fine-grained dynamic management and control of the workflow in operation.

[0061] The process control module is used for fine-grained dynamic management and control of running workflows, supporting four types of operations: pause, terminate, resume, and restart. Its mechanism varies depending on the operation type: When a user initiates a pause or terminate command, the system first reads the task metadata—which includes the task ID, global status (such as running / paused / failed), status of each node (pending / running / success / failed), scheduler type (SLURM / PBS / local), job ID of each running node (such as job_id for SLURM) or local process PID, working directory path, and audit information (submitting user, hostname, operation history, etc.)—to identify the current running context, and depending on the scheduler type, call cancel (SLURM), qdel (PBS), or send SIGTERM / SIGKILL signals to the operating system process to immediately stop the calculation and release resources, while updating the global status to paused or stopped and recording the audit log. The resume operation is only effective when the task is in the paused state. The system does not clear completed results, but instead clears the pause markers and resumes execution from the last incomplete node, maintaining consistency with the original parameters and data. The rerun operation creates a new task instance labeled "Rerun," retains references to the original task, clears the status markers of all nodes (e.g., deleting success / failed files), and starts execution afresh from the beginning, optionally reusing the original input. The entire process accurately reflects the running status and process identifier through task metadata, ensuring that all control operations are precise and traceable. Only "pause" and "terminate" involve job cancellation or process interruption, while "resume" and "rerun" are state reset operations and do not trigger resource reclamation.

[0062] 6. Log Management Module When a task enters the "submitted" state—that is, the user has completed the process configuration and triggered the execution request, and the system has successfully passed dependency checks, generated a DAG, assigned a task ID, and registered the task to the scheduling queue (regardless of whether SLURM, PBS, or local mode is used), but has not yet started executing any tool nodes—the log management module automatically creates a main log file named after the task number (e.g., task_262848b7) (e.g., task_262848b7.log) to continuously append and record task-level events, including status changes (e.g., "submitted → running → completed"), key operations (e.g., "pause", "terminate", "resume"), timestamps, and audit information (operator, hostname, etc.). During process execution, each tool node has a dedicated results directory (e.g., / workspace / task_262848b7 / node_multi_hisat2 / ), which is not only the storage location for output files (e.g., BAM, counting matrix) but also the storage directory for all log files of that node. The system generates various log files in this directory, typically including: node.log: node internal execution log (such as parameter parsing, path conversion, script generation process), stdout.log (standard output stream of tool commands) and stderr.log: standard output error stream of tool commands, rc (exit code file (used to determine success / failure)) and tool_command.sh (the actual submitted job script).

[0063] In addition, task metadata is persistently stored in the task directory, including key fields such as status, start and end time, scheduling type, DAG verification result, and whether to rerun, supporting efficient list query and status verification.

[0064] The log management module not only meets the needs of real-time monitoring and fault diagnosis during operation, but also provides a complete data foundation for subsequent performance analysis, resource utilization statistics and process optimization.

[0065] The above process control module and log management module are both components of the task management subsystem.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A sample batch processing scheduling system based on a workflow engine, characterized in that, The system includes a tool building module, a process orchestration module, a dependency verification module, a process execution module, a process control module, and a log management module; The tool build module is used to configure tool parameters, resource requirements, input / output specifications, generate command lines, and build the runtime environment; The process orchestration module is used to define process content, while converting the tools built by the tool building module into nodes of a directed acyclic graph and establishing dependencies between tools. The dependency verification module constructs a directed acyclic graph and performs multi-stage legality checks to ensure that the process scheduling logic is acyclic and executable. The process execution module schedules tool nodes and executes process tasks based on the node dependencies defined in the directed acyclic graph. The process control module is used to manage and control running process tasks, including pausing, terminating, resuming, or restarting tasks; The log management module creates log files named after the task number, and appends records of status changes, key operations, and timestamps throughout the process; Meanwhile, during the process execution, an independent log is generated for each tool node.

2. The system according to claim 1, characterized in that, The tool building module includes a tool metadata definition unit, a resource configuration unit, an IO specification definition unit, a command line execution unit, and an environment building unit; The tool metadata definition unit is used to configure tool parameters, which include tool name, tool version, tool category, and tool description. The resource configuration unit is used to configure the resource requirements parameters of the tool. The resource requirements parameters include quantitative parameters such as the number of CPU cores, memory capacity, number of GPUs, and maximum runtime. The IO specification definition unit is used to set the input and output specifications, defining the input file paths required by the tool and the expected output file paths; The command-line execution unit is used to generate executable shell commands; The environment building block is used to build the environment in which the tool runs.

3. The system according to claim 2, characterized in that, The command-line execution unit replaces all placeholders in the template with actual values, generating a complete shell executable script.

4. The system according to claim 1, characterized in that, The process orchestration module includes a process definition unit, a tool node generation unit, and a parameter dependency definition unit; The process definition unit is used to set the process name, version number, process description field and process category; the process name is used to identify the unique logical name of the process, the version number is used to track process iteration, the process description field is used to describe the purpose, inputs and outputs and applicable scenarios of the process, and the process category is used to identify the application area or analysis type of the process. The tool node generation unit instantiates the tools built by the tool building module into nodes of a directed acyclic graph; The parameter dependency definition unit establishes the dependency relationship between tools through the parameter passing rules between parent and child nodes.

5. The system according to claim 4, characterized in that, Instantiating tools as nodes in a directed acyclic graph involves identifying and parsing all tools selected by the user through the tool building block; and instantiating each tool as a node with specific attributes based on the tool parameters. Each node contains all the information required to execute the tool, including command-line scripts, environment variable settings, and resource requirements. The parameter passing rules include: the parent node is the tool node that generates the output result after execution, and the child node is the subsequent tool node that references the output result of the parent node through predefined mapping rules for the input parameters.

6. The system according to claim 1, characterized in that, The dependency verification module constructs a directed acyclic graph and performs multi-stage validity checks to ensure that the process scheduling logic is acyclic and executable, including: Self-loop check: If a node has an edge pointing to itself, an error is immediately reported; Cycle dependency detection: Using depth-first search, by maintaining the set of visited nodes and the current recursive path stack, it detects whether there is a closed path. If a closed path is found, the specific cycle path is returned. Sub-process validation: The user specifies a start node and an end node. All intermediate nodes reachable from the start node and reachable from the end node are extracted, forming a logically closed subgraph. Then, forward and reverse reachability validation is performed, including traversing all reachable nodes along the dependency direction from the start node, denoted as the ForwardSet; and traversing all predecessor nodes along the reverse dependency direction from the end node, denoted as the BackwardSet. The subgraph node set is then the intersection of the ForwardSet and BackwardSet. Finally, the existence of isolated nodes is checked. Isolated nodes are nodes that exist between the start and end nodes but are not in the subgraph node set. Verification results and execution guarantees: If the verification fails, a structured error message is returned; if the verification passes, a valid directed acyclic graph is output as the scheduling basis for the process execution module.

7. The system according to claim 1, characterized in that, The process execution module schedules tool nodes based on node dependencies to execute process tasks, including: Load process metadata and all node configurations, and skip user-marked skip nodes; Create a dedicated working directory for each node that is not skipped and pre-generate a shell script; Check the dependency satisfaction of the node. If the dependency is satisfied, all input parameters of the node have obtained valid values ​​and all parent nodes have completed successfully. Before any node is executed, the system parses all input parameters of that node in real time and replaces the placeholders in the pre-generated Shell script with the actual values ​​corresponding to the input parameters. Submit the replaced Shell script and configure the scheduler type; Poll the scheduler status and read the rc file in the node's working directory to determine the execution result; the rc file contains the command exit code. If the command exit code is 0, the task execution is successful; otherwise, the task execution fails. If the task execution fails, immediately trigger a global failure, terminate all running nodes, and record the error log. For nodes that execute successfully, their output parameters are registered in the global context; subsequent nodes automatically retrieve the latest values ​​before triggering. After a process task is successfully executed, the output is a collection of business result files generated by the final node, while the task metadata and complete operation logs are persisted. When a process task fails to execute, the error information, rc value and context snapshot of the first failed node are returned for user diagnosis.

8. The system according to claim 1, characterized in that, The process control module is used to manage and control the running process tasks; When pausing or terminating a task, first read the task metadata to identify the context in which the current task is running; Then, depending on the scheduler type, a job cancellation command is invoked or an interrupt or forced termination signal is sent to the operating system process to stop computation and release computing resources, while updating the global state of the task. When resuming a task, retain the completed results, clear the pause marker, and then continue execution from the last unfinished node; When rerunning a task, a new task instance with the Rerun tag is created, the references to the original task are preserved, the status flags of all nodes are cleared, and the process task is executed from the beginning.

Images