Sample level task scheduling method and system based on placeholder command line template

Abstract

This invention relates to a sample-level task scheduling method and system based on placeholder command-line templates, belonging to the field of automated bioinformatics processing technology. It includes: automatically identifying and isolating samples from a user-provided data file containing sample metadata; designing a declarative command generation mechanism based on placeholder templates to generate executable shell commands for each independent sample; generating independent subprocesses for each sample's shell commands, running multiple tasks in parallel; for samples whose tasks have successfully executed, calculating new field values ​​for the sample based on user-defined new metadata columns and value templates, and appending them to the original sample record; and merging all samples after processing to generate a unified output file. This invention, through declarative placeholder templates and an automated execution mechanism, allows researchers to construct complex batch bioinformatics analysis workflows without writing any code, significantly lowering the technical threshold.

Description

Technical Field

[0001] This invention belongs to the field of automated processing technology in bioinformatics, and relates to a bioinformatics sample-level task scheduling method and system based on placeholder command line templates. Background Technology

[0002] In bioinformatics data analysis (such as whole-genome sequencing, RNA-seq, ChIP-seq, etc.), researchers often need to perform highly similar standardized analysis processes on hundreds or thousands of biological samples, such as alignment, deduplication, variant detection, or expression quantification. To address this need, the industry currently mainly uses the following technical methods: First, manually executing command-line tools line by line, where users sequentially enter commands such as `bwa mem`, `samtools sort`, or `STAR --runThreadN` in the terminal to process each sample. While intuitive, this method is extremely inefficient, prone to parameter errors or path confusion due to human error, and lacks systematic management of intermediate results and metadata, making it almost impractical with large sample volumes. Second, writing customized scripts (such as Bash loops or Python scripts) to automatically construct and execute commands by reading the sample list. This method can achieve a certain degree of automation, but requires users to have programming skills. More importantly, different projects often require redeveloping scripts, resulting in poor code reusability, high maintenance costs, and a lack of unified input / output standards, easily leading to working directory chaos, file overwriting, or untraceable results.

[0003] Furthermore, some researchers have turned to professional workflow management systems such as Snakemake, Nextflow, or Cromwell. These systems are powerful, based on declarative languages ​​to define analysis processes, and support dependency resolution, breakpoint resumption, containerized deployment, and cluster scheduling. However, their configuration process is complex, with a steep learning curve, often requiring the writing of dedicated process description files (such as Snakefile or Nextflow DSL), posing a significant barrier to use for biologists who primarily conduct wet experiments and lack software engineering backgrounds. Moreover, these systems are designed to support complete, complex, multi-step analysis pipelines, which seems overkill for lightweight tasks that only require calling one or two command-line tools, increasing deployment and debugging overhead. Another alternative is commercial or graphical analysis platforms, such as Galaxy, CLC Genomics Workbench, or BaseSpace. These platforms lower the operational threshold through visual interfaces, but generally suffer from strong closedness and limited scalability—difficult to flexibly integrate the latest open-source command-line tools and insufficient support for dynamic metadata updates. Crucially, these platforms typically store data in internal databases or specific formats, making it difficult to seamlessly integrate with local file systems and limiting their application in local high-performance computing environments.

[0004] In summary, the problems with existing technologies are that all current mainstream solutions have failed to achieve an effective balance between "ease of use", "universality" and "automation". Specifically, they are: (1) highly dependent on the user's programming ability, even if only standard tools need to be run in batches, scripts often need to be written or modified; (2) lacking a truly "zero-code" universal sample loop mechanism, it is impossible to achieve automatic task distribution across samples by simply using parameterized command templates combined with structured sample tables (such as CSV / JSON); (3) the analysis results are disconnected from metadata, and key information such as newly generated file paths and status flags need to be manually recorded or sorted out later, which is prone to errors and hinders downstream integration analysis; (4) the sample isolation mechanism is weak, and multiple samples often share the same workspace, which poses a risk of filename conflicts or intermediate result pollution. Although general parallel tools such as GNU Parallel and xargs can accelerate command execution to a certain extent, they do not understand sample semantics, cannot automatically create sample-specific directories based on structured input, dynamically replace path placeholders, and do not support writing execution results back to the metadata table.

[0005] Therefore, existing technologies cannot meet the automated analysis paradigm of "sample as the basic processing unit" in bioinformatics, and there is an urgent need for a new processing method that is lightweight, declarative, requires no programming, and can automatically maintain the metadata link. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a sample-level task scheduling method and system based on placeholder command line templates. By introducing a declarative instruction mechanism based on placeholder templates, it is possible to automatically call any Linux command-line bioinformatics tools without writing code. This is suitable for the need for standardized processes of parallel execution of multiple samples in bioinformatics data analysis, thereby improving the standardization, repeatability and execution efficiency of the analysis process.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A sample-level task scheduling and data update method based on placeholder command line templates, comprising:

[0009] Automatically identify and isolate samples from user-provided data files containing sample metadata.

[0010] Design a declarative command generation mechanism based on placeholder templates to generate executable Shell commands for each independent sample;

[0011] Each sample's shell command generates an independent subprocess, allowing multiple sample tasks to run in parallel.

[0012] For samples where the task is successfully executed, new field values ​​are calculated based on user-defined new metadata columns and value templates, and then appended to the original sample record.

[0013] After all samples have been processed, they are merged to generate a unified output file.

[0014] Furthermore, sample isolation includes: parsing each line of the input data file into an independent sample, and then creating a dedicated working subdirectory for each sample based on a user-specified unique identifier column;

[0015] When creating a working subdirectory for each sample, if the same working subdirectory already exists, a version suffix will be appended to the later-created working subdirectory; if the creation of the working subdirectory fails, the execution will be terminated immediately and an error code will be returned.

[0016] Furthermore, a declarative command generation mechanism based on placeholder templates is designed to generate executable shell commands for each independent sample, including:

[0017] The user inputs a Shell command template containing structured placeholders. All placeholders are matched using regular expressions. Then, the corresponding values ​​are extracted from the current sample metadata according to the semantics of the placeholders and the placeholders in the Shell command template are replaced. Finally, the replaced strings are concatenated to obtain the executable Shell command, and the completeness and validity of the executable Shell command are checked.

[0018] Furthermore, the placeholders include {d}, {col_name}, {i:col:idx}, {b:col:idx}, and {s:col:idx}; where {d} is replaced with the absolute path of the current sample's working directory, {col_name} is replaced with the path value entered in the column named col_name for this sample, {i:col:idx} is replaced with the idx-th element extracted from the semicolon-separated multi-value string, {b:col:idx} is replaced with the complete filename retained based on the extraction of the idx-th element, and {s:col:idx} is replaced with the main filename without the extension retained based on the extraction of the idx-th element.

[0019] Furthermore, after a sample task is completed, it is checked whether the exit code of its corresponding Shell command is 0, and whether the target file implicitly or explicitly dependent in the user logic actually exists. If both conditions are met, the sample task is determined to have been successfully executed.

[0020] Furthermore, based on the user-defined new metadata columns and value templates, new field values ​​for the samples are calculated and appended to the original sample records. This includes, after the sample task is successfully completed, performing secondary placeholder parsing on the template based on the current sample context and saving it as a structured field mapping table containing metadata for each sample. When the new field value is calculated, the structured field mapping table is updated in the form of key-value pairs of "field name → value". If the field name already exists, the original value is overwritten; otherwise, the field is added.

[0021] On the other hand, the present invention provides a sample-level task scheduling and data update system based on placeholder command line templates. The system includes an input layer, a data file parsing module, a command generation module, a task scheduling and data update module, and an output layer.

[0022] The input layer is used to receive data files input by the user and provide a command input window for the user;

[0023] The data file parsing module performs sample identification and isolation based on the data file received from the input layer and the instruction input;

[0024] The command generation module generates Shell commands based on the identified samples and checks the completeness and validity of the Shell commands.

[0025] The task scheduling and data update module encapsulates the Shell command for each sample into an independent subprocess, executes the sample tasks in parallel, and records the status of each task in real time; after all samples have been processed, all samples are merged to generate a unified output file.

[0026] The output layer is used to output and display the output file.

[0027] Furthermore, the data file parsing module includes a metadata parsing unit and a directory management unit; the metadata parsing unit is used to read the data file input by the user, identify its format type, and parse each line in the data file into an independent sample; the directory management unit creates a dedicated working subdirectory for each parsed sample according to the sample identifier column specified by the user.

[0028] Furthermore, the command generation module includes a placeholder parsing unit and a command verification unit; the placeholder parsing unit replaces the placeholders in the placeholder template with the actual values ​​corresponding to the sample based on the placeholder template input by the user, and then concatenates the replaced strings into a Shell command; the verification unit is used to perform integrity and legality checks on the Shell command generated by the placeholder parsing unit.

[0029] Furthermore, the task scheduling and data update module includes a task scheduling unit, a status tracking unit, and a metadata update unit. The task scheduling unit traverses all samples, encapsulates the shell command of each sample into an independent subprocess, and creates a task queue to execute sample tasks in parallel. The status tracking unit is used to record the status of each task in real time. The metadata update unit is used to calculate new field values ​​for successfully executed sample tasks, append them to the original sample record, and merge all samples to generate a unified output file after all samples have been processed.

[0030] The beneficial effects of this invention are as follows: By using declarative placeholder templates and an automated execution mechanism, this invention enables researchers to construct complex batch bioinformatics analysis workflows without writing any code, significantly lowering the technical threshold. This invention can be applied to any Linux command-line tool and is suitable for various application scenarios such as FASTQ processing, VCF annotation, and expression quantification, possessing high versatility and scalability. By processing input data files using the method provided by this invention, each sample can be assigned an independent working directory, effectively achieving sample isolation, avoiding file cross-contamination, and ensuring the reliability of analysis results. Furthermore, metadata such as file paths and statuses newly generated during task execution can be automatically written back to the output file, establishing a complete data processing chain, facilitating downstream integration and traceability. The system has a built-in concurrency control mechanism, supporting sample-level parallel execution, efficiently utilizing multi-core server resources, and improving overall processing performance. Simultaneously, this invention can handle multiple input formats such as JSON, CSV, TSV, and TXT, and can automatically handle common boundary cases such as BOM, null values, and semicolon-separated multi-value fields, enhancing the robustness and practicality of the system.

[0031] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0033] Figure 1 This is a schematic diagram of the sample task execution process;

[0034] Figure 2 This is a schematic diagram of the sample-level task scheduling and data update system structure based on placeholder command line templates provided by the present invention. Detailed Implementation

[0035] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0036] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0037] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0038] To address the problems of existing bioinformatics analysis workflows, such as reliance on script writing, poor reusability, low automation, and complex parallel processing, this invention provides a sample-level task scheduling and data update method based on placeholder command line templates, as described below:

[0039] I. An input data-driven sample recognition mechanism is adopted to automatically identify and isolate samples from user-provided data files containing sample metadata.

[0040] Specifically:

[0041] First, the user specifies a data file containing sample metadata (supports JSON, CSV, TSV, and TXT formats). The system reads the file through the metadata parsing unit and identifies its format type. For example, the format type of the data file can be identified based on features such as the BOM header and delimiters.

[0042] Then, each line in the data file is parsed into an independent sample;

[0043] Finally, based on the unique identifier column (such as sample_id) specified by the user via the -d parameter, a dedicated working subdirectory is created for each sample in the output root directory.<base_out_dir> / <sample_id> The `-d` parameter is a string command-line argument option used to explicitly specify which column in the input sample table serves as the unique identifier (i.e., the "ID column") for each sample. Through the `-d` parameter, a traceable and conflict-free workspace is established for each sample, and results are automatically aggregated by sample ID for easy subsequent integration and querying. After parsing the input file (e.g., samples.csv), the system reads the value of the `-d` parameter (e.g., `-d sample_id`) and extracts the corresponding field value (e.g., "S001") from each sample record. This value is used as the subdirectory name to create an isolated working directory under the output root directory (default is output / or specified by `-o`). If the directory already exists, a version suffix (e.g., S001_v2, S001_v3, etc.) is automatically appended to avoid overwriting. If directory creation for any sample fails due to insufficient file system permissions, insufficient disk space, or other I / O errors, the system will immediately terminate the entire analysis process (rather than just skipping the current sample) and return the standardized error code ERR_DIR_CREATE. The design ensures that all samples can operate under a consistent and secure directory structure, preventing results chaos or data contamination caused by abnormal environments of some samples, thereby achieving strict sample isolation and standardized organization.

[0044] Second, a declarative command generation mechanism based on placeholder templates is designed. Users do not need to write loop scripts; instead, they are provided with a command-line template containing structured placeholders. The system uses a placeholder parsing unit to dynamically replace the placeholders in the template with the actual values ​​corresponding to the current sample, thereby generating executable shell commands for each sample.

[0045] Users pass a Shell command template containing structured placeholders via the `-cmd` parameter. The `-cmd` parameter is a string-based command-line argument option that can be reused multiple times. Users can define multiple sequentially executed Shell commands for the same sample by specifying multiple `-cmd` parameters in the command line (e.g., `-cmd "command1" -cmd "command2"`), allowing for the dynamic injection of custom processing logic during task execution. The purpose of this parameter is to allow users to declaratively embed any external scripts, tools, or processes into the system workflow as templates, flexibly extending task behavior without modifying the core system code.

[0046] When parsing command lines, the system collects all `-cmd` parameter values. Each command template is processed by an internal placeholder parsing unit—this unit identifies structured placeholders in the template (such as `{d}`, `{col_name}`, `{i:col:idx}`, etc.), performs semantic parsing and value replacement based on the metadata context of the current sample, and ultimately generates one or more complete, executable shell commands, which are then executed sequentially in a child process dedicated to that sample. Specifically, the system performs the following operations through the placeholder parsing unit:

[0047] First, the regular expression `\{(?:d|[^{}:]+|(?:[ibs]:[^{}:]+:\d+))\}` is used to match all structured placeholders. This pattern can precisely match placeholders in the form of `{d}`, `{col_name}`, `{i:col:idx}`, `{b:col:idx}`, `{s:col:idx}`, etc. The purpose of placeholder matching is to extract all variable markers that need to be dynamically replaced from the user-provided command template, preparing for subsequent context-based data population.

[0048] Then, based on the placeholder semantics, the corresponding values ​​are extracted from the current sample metadata, and the replacement is performed. After loading the data file, the system constructs a metadata context containing the values ​​of each column field for each row (i.e., a sample). The semantics and replacement rules of each placeholder are as follows:

[0049] (1) {d}: Replace with the absolute path of the working directory unique to the current sample (created dynamically by the system for this sample);

[0050] (2) {col_name}: Replace with the path value entered in the column named col_name for this sample; if the value is a relative path, it will be automatically converted to the absolute path of the file in the Linux file system;

[0051] Here, "column" refers to a field column in a user input data file (such as CSV, TSV, or a tabular configuration file). Each column has a specific column name (such as fastq_path, sample_id, etc.) and represents a certain type of sample attribute or input path.

[0052] (3) {i:col:idx}: For the multi-valued strings (e.g. / a.fq; / b.fq) stored in the col column separated by semicolons (;), extract the idx-th element (counting from 0) and replace the entire placeholder with the value of this element;

[0053] (4) {b:col:idx}: Based on the extraction of the idx-th element, further retain its complete file name (including extension), such as "sample_1.fastq.gz", and replace the placeholder {b:col:idx} with the complete file name;

[0054] (5) {s:col:idx}: Based on the extraction of the idx-th element, only the main file name without the extension is retained, such as "sample_1", and the placeholder {s:col:idx} is replaced with the main file name without the extension.

[0055] Finally, after replacing all placeholders, the strings are concatenated into a complete executable shell command string, which is then passed to the command verification unit for validation of each sample shell command string. The command verification unit does not restrict the physical location or access range of the path; instead, it only checks the integrity and validity of the final generated shell command string. Specifically, it verifies whether the command is a structurally complete single-line shell instruction, ensuring that it does not contain unclosed quotes, newline characters, logical operators (&&, ||), or other dangerous character sequences that could lead to command concatenation or injection. If a command is found to have syntax breaks, potential injection points, or unexpected control characters, execution is refused and a security error is returned. This ensures the security and atomicity of command execution while allowing users to freely specify paths.

[0056] Third, construct a sample-level loop and controllable parallel execution architecture. The system creates a task scheduling queue, iterates through all samples, encapsulates each sample into an independent subprocess, and submits it to a thread pool for execution.

[0057] Specifically:

[0058] First, the system loads all sample records from the input data, combines them with the command template provided by the user via the -cmd parameter, and generates a specific Shell command sequence for each sample after parsing the placeholders. Then, it creates an independent child process for this command sequence (e.g., using fork() + exec() or an advanced process management library), and submits the execution task of this child process to the thread pool for scheduling.

[0059] Secondly, users can set the maximum concurrency using the `-m` parameter (e.g., `-m 4`). The `-m` parameter is an integer command-line argument option that specifies the maximum number of sample tasks the system allows to run simultaneously. This parameter dynamically controls system resource usage, preventing memory exhaustion or I / O overload due to excessive concurrency. Its working principle is as follows: the thread pool maintains an active task slot with a capacity of `-m`. When the number of running tasks reaches this limit, subsequent tasks are temporarily stored in a waiting queue. Once any task completes and releases a slot, the scheduler immediately retrieves the next task from the waiting queue and starts execution, thus achieving smooth and controllable parallel processing.

[0060] Under this architecture, the status of each task (pending execution, running, successful, failed) is recorded in real time by the status tracking unit, and the failure of any task does not affect the scheduling of other samples.

[0061] IV. Implement automatic write-back and output management of result metadata. When a sample task is completed, check whether the command exit code and target file have been successfully generated. If successful, automatically calculate the new field value according to the user's predefined rules and append it to the original sample record.

[0062] Specifically:

[0063] Users define the new metadata columns to be written back and their value templates using the -u or -uc parameters (e.g., -uc bam_path={d} / aln.bam). Both -u and -uc are string-based command-line options used to control how task results are fed back to the sample metadata.

[0064] (1) -u output.tsv: Specifies an output file name (such as results.tsv), indicating that the system should add new fields and generate an updated text file based on the original input sample table (usually in TSV / CSV format) provided by the user through the -i parameter.

[0065] (2) -uc field_name=template: Defines a new field field_name and its value template (e.g., bam_path={d} / aln.bam). The template can contain structured placeholders (e.g., {d}, {col_name}, {i:col:idx}, etc.). This field will be added to the output text file specified by -u above.

[0066] The `-u` and `-uc` parameters work as follows: After a task is successfully completed, the system performs secondary placeholder parsing on the value template based on the current sample's context (including the working directory, original input column values, etc.) and saves it in memory as a structured field mapping table (such as a dictionary or JSON object) containing metadata for each sample. When a new field value is calculated (e.g., in ` / output / S001 / aln.bam`), the structured field mapping table is updated in key-value pairs of "field name → value"—if the field name already exists, the original value is overwritten; if it does not exist, the field is added. Finally, all structured field mapping tables for all sample data are serialized into an output file (e.g., `results.csv`) to ensure strict alignment between the metadata and the task results.

[0067] When a sample task is completed, the system first checks whether its Shell command exit code is 0 and whether the target file implicitly or explicitly dependent on in the user logic (i.e., the key output path derived from the placeholder in the command template, such as {d} / aln.bam) actually exists. If both conditions are met, the sample task is determined to have been executed successfully.

[0068] Fifth, it provides a zero-code, single-command operation mode and a secure preview mechanism. The entire process can be completed with just one command, achieving true "zero-code" operation.

[0069] Specifically:

[0070] The system provides a dry run mode (-n parameter), where -n is a boolean command-line flag (no value required). In this mode, only the directory creation and command generation process is simulated; no actual operations are performed, and a complete execution plan is output for the user to preview.

[0071] In addition, the system provides an input format adaptation mechanism. For example, when inputting JSON format data files, the system automatically disables placeholders such as {i:col:idx} that depend on the order of fields to prevent data misalignment caused by the unordered nature of JSON objects. At the same time, all path parsing is validated to ensure execution security and consistency.

[0072] The following is a description of the command-line scheduling system tool implemented based on the method of this invention:

[0073] Usage of . / batch_runner_linux_3.0:

[0074] -cmd value

[0075] Command template to execute (can be used multiple times)

[0076] -d string

[0077] Column name for sample ID (required)

[0078] -hShow usage information and examples

[0079] -i string

[0080] Input file path (required)

[0081] -if string

[0082] Force input file format (json / csv / tsv / txt)

[0083] -ifs string

[0084] Separator for multi-value input fields (default ";")

[0085] -m int

[0086] Max parallel workers. If less than 1, processes samples sequentially

[0087] -nDry run mode (preview actions)

[0088] -o string

[0089] Base output directory (default ".")

[0090] -of string

[0091] Specify output format for update file (json / csv / tsv / txt)

[0092] -u string

[0093] Path template for update output file (may contain {d})

[0094] -uc value

[0095] Update column definition (-uc ColName=Template), can be used multipletimes

[0096] Examples:

[0097] # Basic usage with JSON input:

[0098] . / tool -i data.json -d sample_id -cmd "cp {fastq1} {d} / "

[0099] # Processing with multiple commands:

[0100] . / tool -i samples.csv -d id -cmd "mkdir -p {d}" -cmd "cp {data_file}{d} / " -cmd "chmod +x {d} / " -u results.tsv -uc status=done -uc log_path={d} / log.txt -m 4

[0101] # Processing with per-sample update files using {d} in -u:

[0102] . / tool -i samples.csv -d sample_id -cmd "run_analysis.sh {input}" -u{d} / analysis_results.csv -uc result_status=complete

[0103] #Dry run to see what would happen:

[0104] . / tool ​​-i data.json -d sample_id -cmd "ls {d}" -n

[0105] # Using {s:col:idx} to get basename without extension:

[0106] . / tool ​​-i samples.csv -d id -cmd "cp {i:files:0} {d} / {s:files:0}.bak"

[0107] The method of the present invention will be specifically illustrated below through two embodiments.

[0108] Example 1: RNA-seq multi-sample alignment and automatic BAM path rewriting.

[0109] The user intends to align three RNA-seq samples and generate a sorted BAM file, while automatically recording the result path. The user needs to perform the following operations:

[0110] 1. Prepare the data file, specifically a CSV format sample metadata file named samples.csv, with the following content:

[0111] sample_id, fastq1, fastq2

[0112] S001, / data / fastq / S001_R1.fq.gz, / data / fastq / S001_R2.fq.gz

[0113] S002, / data / fastq / S002_R1.fq.gz, / data / fastq / S002_R2.fq.gz

[0114] S003, / data / fastq / S003_R1.fq.gz, / data / fastq / S003_R2.fq.gz

[0115] 2. Execute the following single command in the terminal:

[0116] / src / batch_runner_linux_3.0 \

[0117] -i sample.csv \

[0118] -d sample_id \

[0119] -cmd "hisat2 -p 4 -x / ref / hg38 -1 {fastq1} -2 {fastq2} | samtoolssort -o {d} / align.bam" \

[0120] -u result.csv \

[0121] -uc bam_path={d} / align.bam \

[0122] -m 6

[0123] It is important to note that the "zero-coding" mentioned in this invention does not mean that no text input is required at all. Rather, it means that users do not need to write any scripts, functions, loops, or flow control code. They can fully express their analysis intent simply through a declarative command that conforms to the semantics of the natural task. For example, parameters such as -cmd, -u, and -m are directly concatenated by the user in a natural language style in the command line, which is essentially a high-level declaration of the task logic.

[0124] (1) -cmd is followed by a standard Shell command template familiar to the user (such as hisat2). You only need to replace the specific path in the original manually run command with a structured placeholder (such as {fastq1}), without writing any scheduling, loop or error handling code.

[0125] (2) -uc and -u are used to declare "I want to record the path of the alignment results into a new column bam_path and output it to results.csv", which is equivalent to filling in the column names and rules of a spreadsheet, rather than programming;

[0126] (3) -m 4 simply means "a maximum of 4 samples can be run at the same time", analogous to the "concurrency slider" in the graphical interface.

[0127] Therefore, the user's operation is equivalent to "telling the system what to do in one sentence". The system automatically completes all engineering details such as sample splitting, path binding, parallel scheduling, and result writing back, realizing bioinformatics analysis with "zero scripts and zero process code".

[0128] Based on the single command mentioned above, the system automatically completes the following full-process handling according to the following steps:

[0129] S1. Sample Identification and Directory Creation: The metadata parsing module reads samples.csv and identifies three rows of samples; the directory management unit creates output / S001, output / S002 and output / S003 folders in the current directory (the default output directory is output).

[0130] S2. Dynamic Command Generation: Taking S001 as an example, {fastq1} and {fastq2} will be replaced with the actual paths in the corresponding columns, and {d} will be automatically replaced with the sample-specific directory such as / output / S001. The system dynamically generates and prepares the actual command for execution.

[0131] bwa mem -t 4 / ref / hg38.fa / data / S001_R1.fq.gz / data / S001_R2.fq.gz | \

[0132] samtools sort -o / output / S001.aln.bam

[0133] Finally, the command verification unit performs integrity and validity checks on the generated Shell command string.

[0134] S3. Parallel Sample Scheduling: The system starts a task scheduler that iterates through all sample records. For each sample, its specific command is encapsulated into an independent subprocess and submitted to the execution queue.

[0135] The `-m 4` parameter limits the maximum concurrency to 4 to prevent CPU or memory overload. If the number of samples exceeds the concurrency limit, subsequent tasks will be queued to ensure resource control and system stability.

[0136] S4. Metadata Write-back: When a command for a sample is successfully executed, the system calculates the bam_path value (e.g., / output / S001 / aln.bam) of the sample based on the new column template defined by -uc bam_path={d} / aln.bam, and inputs it as a new field into the new sample structure metadata file. If the sample fails to match due to FASTQ corruption, its bam_path field is left blank, while the other fields are retained.

[0137] Finally, after all samples have been processed, they are merged to generate a unified new sample structure metadata file, results.csv, with the contents shown below. This automatically maintains the "input-output" metadata link, facilitating downstream analysis, quality control, or database entry.

[0138] sample_id,fastq1,fastq2,bam_path

[0139] S001, / data / fastq / S001_R1.fq.gz, / data / fastq / S001_R2.fq.gz, / output / S001 / aligned.bam

[0140] S002, / data / fastq / S002_R1.fq.gz, / data / fastq / S002_R2.fq.gz, / output / S002 / aligned.bam

[0141] S003, / data / fastq / S003_R1.fq.gz, / data / fastq / S003_R2.fq.gz, / output / S003 / aligned.bam

[0142] S5. Security and Output: The entire process requires only a single command to complete all operations from input parsing and parallel execution to result integration, truly achieving "zero scripts, zero programming." Furthermore, the system incorporates multiple security measures, including support for a dry run mode (-n) to preview all created directories and executed commands without actually running them. The preview information generated in dry run mode is shown below:

[0143] Dry operation mode commands:

[0144] . / src / batch_runner_linux_3.0 \

[0145] -i next.tab -d group \

[0146] -cmd "mkdir -p {d} / " \

[0147] -cmd "cp {data_file} {d} / " \

[0148] -cmd "chmod +x {d} / \

[0149] -u results.tsv -uc status=done \

[0150] -uc log_path={d} / log.txt \

[0151] -m 4 \

[0152] -if tsv \

[0153] -n

[0154] Dry-run mode operation log:

[0155] Starting processing of 3 samples...

[0156] Number of commands per sample: 3

[0157] Input field separator used: ';'

[0158] Using up to 4 parallel workers.

[0159] --- DRY-RUN Configuration Summary ---

[0160] Base output directory: / home / pengshu / project / deseq2

[0161] Input file:next.tab

[0162] Detected / Forced input format:tsv

[0163] Sample ID column:group

[0164] Number of commands: 3

[0165] Command 1: mkdir -p {d} /

[0166] Command 2: cp {data_file} {d} /

[0167] Command 3: chmod +x {d} /

[0168] Input field separator:;

[0169] Update file template:results.tsv

[0170] Update output format:tsv

[0171] Update columns to be added / calculated:

[0172] status = done

[0173] log_path = {d} / log.txt

[0174] Execution mode: Parallel with 4 workers

[0175] -------------------------------------

[0176] [DRY-RUN NOTE] Update file path does not contain {d}, so a singleaggregate file would be created at: results.tsv

[0177] Dry-run complete. No commands were executed.

[0178] Meanwhile, JSON input protection avoids data misalignment caused by the unordered nature of JSON objects by automatically disabling placeholders such as {i:col:idx} that depend on field order, and the command validity verification function verifies whether the generated command is executable.

[0179] Example 2: Multi-value field processing (ChIP-seq duplicate sample merging).

[0180] The user inputs multiple FASTQ paths, separated by semicolons, as shown below:

[0181] sample_id, fastq_list

[0182] ChIP_A, / data / A_rep1.fq; / data / A_rep2.fq

[0183] ChIP_B, / data / B_rep1.fq; / data / B_rep2.fq

[0184] The single execution command used is as follows:

[0185] / src / batch_runner_linux_3.0 \

[0186] -i chip_samples.csv

[0187] -d sample_id \

[0188] -cmd "cat {i:fastq_list:0} {i:fastq_list:1}>{d} / merged.fq" \

[0189] -uc merged_fastq={d} / merged.fq \

[0190] -u update.tsv

[0191] The system correctly extracts the 0th and 1st file paths using the {i:col:idx} placeholder, completes the merging, and adds a merged_fastq column to the output.

[0192] This invention also provides a sample-level task scheduling and data update system based on placeholder command line templates, such as... Figure 2 As shown, the system includes an input layer, a data file parsing module, a command generation module, a task scheduling and data update module, and an output layer.

[0193] The input layer receives data files from the user and provides a command input window. Specifically, the input layer receives sample metadata data files provided by the user, supporting formats such as CSV, JSON, TSV, and TXT. It also provides an input window for placeholder templates and input windows for runtime control parameters, such as -d, -cmd, -u, -uc, and -n parameters.

[0194] The data file parsing module includes a metadata parsing unit and a directory management unit. The metadata parsing unit identifies the format type of the data file received from the input layer and parses each line of the data file into an independent sample. The directory management unit creates a dedicated working subdirectory for each parsed sample based on the sample identifier column specified by the user.

[0195] The command generation module includes a placeholder parsing unit and a command verification unit.

[0196] The placeholder parsing unit, based on a user-input placeholder template, replaces the placeholders in the template with the actual values ​​corresponding to the samples, and then concatenates the replaced strings into executable shell commands. The placeholders include {d}, {col_name}, {i:col:idx}, {b:col:idx}, and {s:col:idx}. During replacement, {d} is replaced with the current sample working directory path, {col_name} is replaced with the absolute path of that column (relative paths are automatically converted to absolute paths), {i:col:idx} extracts the idx-th element from the semicolon-separated multi-value field, and {b:col:idx} and {s:col:idx} extract the filenames with and without extensions, respectively.

[0197] The command verification unit is used to verify the Shell commands generated by the placeholder parsing unit, and to check their integrity and legality.

[0198] The task scheduling and data update module includes a task scheduling unit, a status tracking unit, and a metadata update unit.

[0199] The task scheduling unit iterates through all samples, encapsulates the Shell command corresponding to each sample into an independent subprocess, and creates a task queue. It executes sample tasks in parallel according to the number of concurrent tasks input by the user, and tasks that exceed the concurrency limit enter the waiting queue.

[0200] The status tracking unit is used to record the status of each task in real time, including status such as pending execution, running, success, failure, etc.

[0201] The metadata update unit is used to calculate new field values ​​for successfully executed sample tasks, and finally merges all samples to generate a unified new metadata structure file.

[0202] In summary, this invention provides a sample-level task scheduling and data update method and system based on placeholder command-line templates. Through declarative placeholder templates and automated execution mechanisms, it enables researchers to construct complex batch bioinformatics analysis workflows without writing any code, significantly lowering the technical threshold. The system is compatible with any Linux command-line tool and is suitable for various application scenarios such as FASTQ processing, VCF annotation, and expression quantification, exhibiting high versatility and scalability. Each sample is assigned an independent working directory, effectively achieving sample isolation, avoiding file cross-contamination, and ensuring the reliability of analysis results. Metadata such as newly generated file paths and statuses during task execution can be automatically written back to the output file, establishing a complete data processing chain for easy downstream integration and traceability. The system has a built-in concurrency control mechanism, supporting sample-level parallel execution and efficiently utilizing multi-core server resources to improve overall processing performance. It also supports multiple input formats such as JSON, CSV, TSV, and TXT, and can automatically handle common boundary cases such as BOM, null values, and semicolon-separated multi-value fields, enhancing the system's robustness and practicality.

[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A sample-level task scheduling method based on placeholder command line templates, characterized in that, The method includes: Automatically identify and isolate samples from user-provided data files containing sample metadata. Design a declarative command generation mechanism based on placeholder templates to generate executable shell commands for each independent sample. Specifically, the user inputs a shell command template containing structured placeholders, and all placeholders are matched using regular expressions. Then, the corresponding values ​​are extracted from the current sample's metadata according to the placeholder semantics and used to replace the placeholders in the shell command template. Finally, the replaced strings are concatenated to obtain the executable shell command, and the completeness and validity of the executable shell command are checked. Each sample's shell command generates an independent subprocess, allowing multiple sample tasks to run in parallel. For samples where the task is successfully executed, new field values ​​are calculated based on user-defined new metadata columns and value templates, and then appended to the original sample record. After all samples have been processed, they are merged to generate a unified output file.

2. The method according to claim 1, characterized in that, Sample isolation involves parsing each line of the input data file into an independent sample, and then creating a dedicated working subdirectory for each sample based on a user-specified unique identifier column. When creating a working subdirectory for each sample, if the same working subdirectory already exists, a version suffix will be appended to the later-created working subdirectory; if the creation of the working subdirectory fails, the entire data file analysis process will be terminated immediately and an error code will be returned.

3. The method according to claim 1, characterized in that, Placeholders include {d}, {col_name}, {i:col:idx}, {b:col:idx}, and {s:col:idx}; where {d} is replaced with the absolute path of the current sample's working directory, {col_name} is replaced with the path value entered in the column named col_name for this sample, {i:col:idx} is replaced with the idx-th element extracted from the semicolon-separated multi-value string, {b:col:idx} is replaced with the complete filename retained after extracting the idx-th element, and {s:col:idx} is replaced with the main filename without the extension retained after extracting the idx-th element.

4. The method according to claim 1, characterized in that, Once a sample task is completed, check whether its corresponding Shell command exit code is 0 and whether the target file implicitly or explicitly dependent in the user logic actually exists. If both conditions are met, the sample task is considered to have been successfully executed.

5. The method according to claim 1, characterized in that, Based on user-defined new metadata columns and value templates, new field values ​​for samples are calculated and appended to the original sample records. This includes a second placeholder parsing of the template based on the current sample context after the sample task is successfully completed, and saving it as a structured field mapping table containing metadata for each sample. After the new field value is calculated, the structured field mapping table is updated in the form of key-value pairs of "field name → value". If the field name already exists, the original value is overwritten; otherwise, the field is added.

6. A sample-level task scheduling system based on placeholder command line templates, characterized in that, The system includes an input layer, a data file parsing module, a command generation module, a task scheduling and data update module, and an output layer; The input layer is used to receive data files input by the user and provide a command input window for the user; the commands input by the user include placeholder templates and run control parameters; The data file parsing module performs sample identification and isolation based on the data file received from the input layer and the instruction input; The command generation module generates Shell commands based on the identified samples and checks the completeness and validity of the Shell commands. The command generation module includes a placeholder parsing unit and a command verification unit. The placeholder parsing unit replaces the placeholders in the placeholder template with the actual values ​​corresponding to the samples based on the user-input placeholder template, and then concatenates the replaced strings into Shell commands. The command verification unit is used to check the completeness and validity of the Shell commands generated by the placeholder parsing unit. The task scheduling and data update module encapsulates each sample's Shell command into an independent subprocess, executes the sample tasks in parallel, and records the status of each task in real time. After all samples have been processed, all samples are merged to generate a unified output file; The output layer is used to output and display the output file.

7. The system according to claim 6, characterized in that, The data file parsing module includes a metadata parsing unit and a directory management unit. The metadata parsing unit is used to read the data file input by the user, identify its format type, and parse each line in the data file into an independent sample. The directory management unit creates a dedicated working subdirectory for each parsed sample based on the sample identifier column specified by the user.

8. The system according to claim 6, characterized in that, The task scheduling and data update module includes a task scheduling unit, a status tracking unit, and a metadata update unit. The task scheduling unit traverses all samples, encapsulates the shell command of each sample into an independent subprocess, and creates a task queue to execute sample tasks in parallel. The status tracking unit is used to record the status of each task in real time. The metadata update unit is used to calculate new field values ​​for successfully executed sample tasks and append them to the original sample record. After all samples have been processed, all samples are merged to generate a unified output file.

Images