A slurm-based bioinformatics workflow management system

Abstract

This invention provides a Slurm-based bioinformatics workflow management system. The module management subsystem stores bioinformatics software commands in a database according to a predetermined format, defining them as modules. These modules can be combined into new modules through serialization or network connections, enabling the rapid construction of complex workflows. The task submission system converts these modules into corresponding task scripts. These scripts are generated in a nested manner, consisting of a main job, sub-jobs, and the smallest job unit. The scripts conform to the Slurm job scheduling system specifications and include a progress report. This workflow management system, combined with front-end and back-end technologies, enables visual interaction, improves the management and utilization efficiency of bioinformatics software commands, and promotes the rapid development of life sciences.

Description

Technical Field

[0001] This invention relates to the field of bioinformatics management technology, and in particular to a bioinformatics workflow management system based on Slurm. Background Technology

[0002] Bioinformatics, as an interdisciplinary field combining biology, computer science, and information technology, has profound significance in the agricultural sector. Through data-driven and intelligent analysis, it provides innovative solutions for agricultural breeding, pest and disease control, resource utilization, and sustainable development. However, with the continuous development of bioinformatics, a series of problems have gradually emerged.

[0003] At the software level, there are numerous analytical tools with extremely complex parameters. The NCBI database alone includes over 2,000 bioinformatics software programs, covering hundreds of sub-fields such as alignment, variant detection, functional annotation, and evolutionary analysis. These tools not only have a vast number of parameters—for example, the BWA alignment tool has over 20 selectable parameters—but also exhibit complex interactions between these parameters. For instance, GATK variant detection requires pre-processing steps such as alignment quality thresholds and base quality calibration. Researchers need to spend considerable time learning how to use these tools and adjusting parameter combinations, which directly leads to a significant decrease in the efficiency of applying the appropriate software on demand.

[0004] In terms of cluster resource management and task scheduling, Slurm, as an open-source cluster management and job scheduling system, aligns closely with the needs of bioinformatics and is widely used by researchers engaged in bioinformatics-related research. Slurm can dynamically allocate computing resources based on user needs and cluster resources, and schedule tasks according to priority, dependencies, and resource requirements, ensuring orderly task execution. However, current Slurm usage requires users to manually write sbatch scripts, which presents a certain technical barrier for non-professional users. Furthermore, each task submission requires effort in preparation steps unrelated to the command itself; for example, some bioinformatics commands may depend on specific environment variables or file system paths, undoubtedly consuming additional time and effort for users.

[0005] In workflow management, bioinformatics workflows refer to organizing multiple steps, tools, and computational tasks in biological data analysis according to a specific logical sequence to form an automated or semi-automated processing flow. The core objective is to efficiently and accurately complete complex biological data analysis tasks through standardization and modularization, while improving the repeatability and scalability of results. However, traditional methods rely on manually writing scripts to connect multiple software programs, resulting in cumbersome processes, error-prone workflows, and poor code reusability, making it difficult for different researchers or projects to share and learn from each other. Furthermore, facing different data scales and analytical needs, parameters and processes need frequent adjustments, lacking flexibility and scalability. Existing workflow engines (such as Nextflow based on Groovy / Kotlin and Snakemake based on Python syntax) support process definition and dependency management, but require users to learn domain-specific languages ​​(DSLs), increasing deployment complexity.

[0006] The website's front-end and back-end technologies encompass the core components for implementing interactive web applications. The front-end uses HTML / CSS to construct the page structure and style, JavaScript and other technologies to achieve dynamic interaction and data rendering, and communicates with the back-end through AJax / Fetch, etc. The back-end uses databases such as MySQL to store data, ultimately realizing a graphical, interactive, and automated website application. Using a website in conjunction with back-end configuration is a basic function of some bioinformatics websites, providing specific bioinformatics analysis functions (such as NCBI's Blast function), but only within the website's preset functions.

[0007] Currently, there is a lack of a bioinformatics workflow management system that is low in technical barriers, highly flexible, and efficient. Summary of the Invention

[0008] Therefore, it is necessary to provide a Slurm-based bioinformatics workflow management system to address the aforementioned technical problems.

[0009] A Slurm-based bioinformatics workflow management system includes: a module management subsystem and a task submission subsystem;

[0010] The module management subsystem is used to generate, combine and manage various modules. Each module of the module management subsystem includes a command script module, a linear process module and a network structure module. The command script module is connected to the linear process module and the network structure module.

[0011] The command script module is used to store bioinformatics software commands and scripts;

[0012] The linear process module is composed of the command script modules connected in sequence according to the execution order;

[0013] The network structure module is composed of each of the command script modules or the linear process modules connected through dependencies;

[0014] The task submission subsystem is used to convert the selected module into a visual task submission page, automatically determine whether there are multiple parallel inputs based on the content of the form input, and further generate and execute a job script that conforms to the Slurm specification.

[0015] In one embodiment, each module generated by the module management subsystem includes template preset information, parameter preset information, variable preset information, and result preset information; wherein,

[0016] The template preset information records the command line parameters of bioinformatics software commands;

[0017] The parameter preset information record explains the template preset information.

[0018] The preset result information is used to indicate the file to be generated;

[0019] The preset variable information is used to indicate the parameters to be passed in when configuring the actual task.

[0020] In one embodiment, the command script module includes environment preset information, address preset information, thread preset information, and code preset information; wherein,

[0021] The preset environmental information records the system environment when bioinformatics software commands are used;

[0022] The preset address information records the command address of the bioinformatics software;

[0023] The thread preset information records the threads to be allocated by bioinformatics software commands;

[0024] The code is preset to a custom script.

[0025] In one embodiment, the linear process module further includes thread preset information;

[0026] The thread preset information records the threads to be allocated by the bioinformatics software commands.

[0027] In one embodiment, the linear flow module and the network structure module further include sub-modules, wherein the sub-module of the linear flow module is a command script module, and the sub-module of the network structure module is either a command script module or a linear flow module.

[0028] The submodule adjusts its template preset information and parameter preset information without changing its content through template revision and parameter revision.

[0029] In one embodiment, the task submission subsystem is further configured to:

[0030] An input form is generated based on preset module variables. The form is used to receive variables input by the user. The input variables include input files and input parameters.

[0031] The system dynamically displays the preset parameter information and preset result information after the variables are filled in, so as to prompt the bioinformatics commands that are actually applied.

[0032] In response to the input variables, the preset parameter information of the linear process module and the network structure module after variable filling is dynamically displayed.

[0033] Upon receiving revision information, the command script module can perform temporary revisions of template presets and parameter presets when acting as a main module or a sub-module, thereby fine-tuning the actual template presets and parameter presets without changing the content of the sub-module.

[0034] In one embodiment, the task submission subsystem allows specified steps or nodes to run before the task is submitted, without requiring the module content to be re-edited.

[0035] In one embodiment, the job script is based on the Slurm specification and adopts a three-level nested structure, including a main job layer, a sub-job layer, and a minimum job unit;

[0036] The main job layer is used to submit subjobs, with one subjob corresponding to one sample;

[0037] The sub-job layer is used to submit the smallest job unit;

[0038] The smallest unit of operation includes bioinformatics software commands and scripts, and is used to specify the number of threads to be allocated.

[0039] In one embodiment, the network structure module is used to generate a corresponding number of minimum job units with dependencies based on the number of nodes when each sub-job layer of the command script module and the linear process module submits a minimum job unit.

[0040] In one embodiment, the minimum job unit includes a command script type minimum job unit and a linear flow type minimum job unit, wherein the command script type minimum job unit contains a single step, and the linear flow type minimum job unit contains multiple sub-steps.

[0041] The aforementioned Slurm-based bioinformatics workflow management system includes a module management subsystem that stores bioinformatics software commands in a database according to a predetermined format, defining them as modules. These modules can be combined into new modules through serialization or network connections, enabling the rapid construction of complex workflows. The task submission system converts these modules into corresponding task scripts. These scripts are generated in a nested manner, consisting of a main job, sub-jobs, and the smallest job unit. The scripts conform to the Slurm job scheduling system specifications and include progress reports. This workflow management system, combined with front-end and back-end technologies, enables visual interaction, improves the management and utilization efficiency of bioinformatics software commands, and promotes the rapid development of life sciences.

[0042] The present invention is implemented by organizing, recording, utilizing, and reusing actual command-line instructions, making it easier for users to understand and use. Users can create modules from any software they install or any scripts they write, or they can use modules shared by other users on the same server. After the module is created, users can completely bypass the Linux environment and complete the entire bioinformatics analysis task on a web page. Attached Figure Description

[0043] Figure 1 A is a schematic diagram of a module of a Slurm-based bioinformatics workflow management system in one embodiment; Figure 1 B is a schematic diagram of the functional architecture of a Slurm-based bioinformatics workflow management system in one embodiment;

[0044] Appendix Figure 2 This is a schematic diagram of the auxiliary system components of the workflow management subsystem in one embodiment;

[0045] Appendix Figure 3 This is a schematic diagram illustrating the deployment of a workflow management subsystem on a server in one embodiment.

[0046] Appendix Figure 4 This is a schematic diagram of an embodiment of the command script module;

[0047] Appendix Figure 5 This is a schematic diagram of an embodiment of a linear process module;

[0048] Appendix Figure 6 This is a schematic diagram of an embodiment of a network structure module. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0050] Example 1

[0051] In this embodiment, please refer to Figure 1 A and Figure 1 B provides a Slurm-based bioinformatics workflow management system, which includes: a module management subsystem 100 and a task submission subsystem 200;

[0052] The module management subsystem 100 is used to generate, combine, and manage various modules. Each module of the module management subsystem 100 includes a command script module 110, a linear process module 120, and a network structure module 130.

[0053] The command script module 110 is used to store bioinformatics software commands and scripts;

[0054] The linear process module 120 is composed of the command script modules 110 connected in sequence according to the execution order;

[0055] The network structure module 130 is formed by connecting each of the command script modules 110 or the linear process module 120 through dependencies;

[0056] The task submission subsystem 200 is used to convert the selected module into a visual task submission page, automatically determine whether there is multiple parallel input based on the content of the form input, and further generate and execute a job script that conforms to the Slurm specification.

[0057] In this embodiment, the Slurm-based bioinformatics workflow management system is divided into two parts: a module management subsystem and a task submission subsystem. Specifically, the module management subsystem converts bioinformatics commands into modules, and the task submission subsystem converts each module into a Slurm script and submits it. The module management subsystem is used to store bioinformatics software commands in a database according to a predetermined format, defining them as modules. It also allows modules to be combined into new modules through serialization or network connections, thereby enabling the rapid construction of complex workflows. Three types of bioinformatics modules are involved: command script modules (Script), linear flow modules (Stream), and network structure modules (Structure).

[0058] In this embodiment, apart from the simplest command script module, each module can be linearly connected to form a new module. At the same time, the command script module and the linear process module can form a more complex network structure module in a dependent manner. However, the network structure module cannot be further combined into a new module. The above design provides as much freedom as possible while preventing the module structure from becoming too complex.

[0059] In this embodiment, the command script module serves as the most basic module unit, storing a single bioinformatics software command; the command script module is composed of multiple command script modules sequentially connected to form a linear process; the command script module or linear process serves as a unit module, and multiple command scripts or linear processes form a network structure through a dependent method, such as a directed acyclic graph.

[0060] The command script module serves as the foundation, containing information such as the name, address, and runtime environment of individual bioinformatics software commands. The runtime environment can be a pre-defined conda environment or a Docker environment. When executing a command, it's equivalent to adding 'source virtual environment&&' before the command to activate the corresponding environment.

[0061] All three types of modules are recorded in the database and can be shared and imported among users. This is achieved by copying the target module and assigning it a new unique identifier. If submodules exist, the submodules or their submodules are copied together, and the identifiers of the submodules are updated. Editing newly imported modules and submodules will not affect the source module. The module copying function has a similar effect: copying a module will copy the target module and all its subordinate modules, and update the relationships between modules.

[0062] When setting the command name for the command script module, you can choose the type of command name. "Normal" is the default bioinformatics software command. You can also choose "python", "R" and "perl" for custom scripts. The relevant content is written in "Usage Code". When generating the Slurm script, the script file is generated and used in the task log page.

[0063] The Slurm-based bioinformatics workflow management system described above achieves low technical barriers (visualized, interactive, non-Linux and non-programming), high degree of freedom (able to create modules from user-installed software or scripts), and high efficiency (module combination, automatic configuration file generation, progress monitoring, and job management). It improves the management and utilization efficiency of bioinformatics software commands, promoting the rapid development of life sciences.

[0064] In one embodiment, each module generated by the module management subsystem includes template preset information, parameter preset information, variable preset information, and result preset information; wherein...

[0065] The template preset information records the command line parameters of bioinformatics software commands;

[0066] The parameter preset information record explains the template preset information.

[0067] The preset result information is used to indicate the file to be generated;

[0068] The preset variable information is used to indicate the parameters to be passed in when configuring the actual task.

[0069] The parameter preset information contains multiple explanation lines, each line containing the separator equal sign "=". The part before the equal sign is the content of the curly braces in the template preset, and the part after the equal sign is the target value. The target value is the specific value or the variable name enclosed in curly braces. If the target value contains curly braces, it can be further interpreted by the variable preset.

[0070] The result preset information contains multiple interpretation lines, each line containing the separator equal sign "=". The part before the equal sign is the content of the curly braces in the template preset, and the part after the equal sign is the target value. The target value is the specific value or the variable name enclosed in curly braces. If the target value contains curly braces, it can be further interpreted by the variable preset information.

[0071] The variable preset information contains multiple explanation lines, each with an equal sign "=" as a separator. The content before the equal sign is the content in curly braces of the template preset, parameter preset, or result preset, and the content after the equal sign is the target value. The target value is a specific value or a question mark "?". If the target value is a question mark, it indicates the parameter that needs to be passed in when the module is executed.

[0072] Template presets, parameter presets, result presets, and variable presets all support adding comments at the end of each line, separated by a hash symbol "#", making it convenient for users to record parameter functions and other important information. In addition to the above information, the module also records the name, multi-level categories, and comments. After the module is created, a unique identifier is automatically generated for data indexing, and the creator, owner, creation time, and update time are automatically recorded.

[0073] In this embodiment, the command script module, linear process module, and network structure module all include common preset information such as template preset information, parameter preset information, result preset information, and variable preset information. This preset information is the foundation for the modularization and visualization of bioinformatics commands, as well as for parameter input and task submission. Among them, the result preset information is mainly used for prompting; in the non-command script modules, the template preset information and parameter preset information also mainly serve as prompts; when using the wildcard '{*}', they can be used to find the input file name to achieve multi-sample task generation.

[0074] To achieve modularity, visualization, and interactivity, and to improve the flexibility and efficiency of bioinformatics commands, in this embodiment, the software parameters are broken down into template preset information, parameter preset information, and variable preset information. To make the software easier for users to understand, result preset information is also added, which is implemented in the same way as the parameter preset information, but is only used for prompting purposes.

[0075] In this embodiment, the template preset information specifies the basic framework of bioinformatics software commands, and variable content is marked with curly braces "{}". Specifically, the template preset information contains multiple lines of content, including "{input}", which is interpreted by the parameter preset (variable passing, i.e., string substitution). The interpreted template is "{input folder} / {name}.fa", which is further interpreted by the variable as ". / ?.fa". The question mark value is converted into an inputtable form on the task submission page to specify the actual input content.

[0076] On the task submission page, three input forms—name, input folder, and output folder—are generated based on the preset variable information. The default values ​​for the form content are those set in the module. Each variable is assigned a random, highly saturated color, which can be changed randomly by clicking the color box. Regardless of whether the form content or the color is changed, the corresponding values ​​in the parameter presets and result presets will change dynamically simultaneously. Temporary revisions to the template presets and parameter presets are possible, taking effect only upon task submission without altering the corresponding module content.

[0077] In one embodiment, the command script module includes environment preset information, address preset information, thread preset information, and code preset information; wherein,

[0078] The preset environmental information records the system environment when bioinformatics software commands are used;

[0079] The preset address information records the command address of the bioinformatics software;

[0080] The thread preset information records the threads to be allocated by bioinformatics software commands;

[0081] The code is preset to a custom script.

[0082] It is worth mentioning that the preset information in each embodiment refers to various parameters and information that are input and stored in the database.

[0083] In this embodiment, the command script module includes environment preset information, address preset information, thread preset information, template preset information, parameter preset information, result preset information, variable preset information, and usage code (such as Python code); the linear flow module includes thread preset information, template preset information, parameter preset information, result preset information, variable preset information, and a sub-module (command script module); the network structure module includes template preset information, parameter preset information, result preset information, variable preset information, and a sub-module (command script module or linear flow module).

[0084] In one embodiment, the linear process module further includes thread preset information;

[0085] The thread preset information records the threads to be allocated by the bioinformatics software commands.

[0086] In one embodiment, the linear flow module or network structure module further includes a sub-module, wherein the sub-module of the linear flow module is a command script module, and the sub-module of the network structure module is a command script module or a linear flow module.

[0087] The submodule adjusts its template preset information and parameter preset information without changing its content through template revision and parameter revision.

[0088] In this embodiment, the first submodule and the second submodule are subordinate modules of the corresponding module and are directly referenced and used through a unique identification code; the submodule can revise the template preset information and parameter preset information, thereby fine-tuning the actual template preset information and parameter preset information used without changing the content of the submodule.

[0089] Template revision rules allow multiple lines of content, each line containing a delimiter "===". The content before the delimiter is the original content in the template preset, and the content after the delimiter is the target value. If the target value is empty, the original content in the template preset is deleted; otherwise, the corresponding target value is added to the end of the template preset. Parameter revision rules also allow multiple lines of content, each line containing a delimiter "=". The content before the delimiter corresponds to the content before the delimiter in each line of the parameter preset, and the content after the delimiter is the target value after the replacement. The module management subsystem and the user system work together to enable module copying, sharing, and referencing of modules shared by other users.

[0090] In one embodiment, the task submission subsystem is further configured to:

[0091] An input form is generated based on preset module variables. The form is used to receive variables input by the user. The input variables include input files and input parameters.

[0092] The system dynamically displays the preset parameter information and preset result information after the variables are filled in, so as to prompt the bioinformatics commands that are actually applied.

[0093] In response to the input variables, the preset parameter information of the linear process module and the network structure module after variable filling is dynamically displayed.

[0094] Upon receiving revision information, the command script module can perform temporary revisions of template presets and parameter presets when acting as a main module or a sub-module, thereby fine-tuning the actual template presets and parameter presets without changing the content of the sub-module.

[0095] In this embodiment, the task submission subsystem can also generate an inputtable form based on module variable presets to specify input variables, including input files and input parameters, allowing multiple sample inputs; the input variables allow the use of the symbol "{*}" as a wildcard, and based on the template presets, parameter presets, and the variable content of the form input, it detects whether there are multiple sample inputs, so as to use for batch submission of multiple sample tasks;

[0096] The task submission subsystem dynamically displays the preset parameter and result information after variable filling, and can prompt the bioinformatics commands that are actually applied. By automatically assigning a high-saturation color to each variable, it dynamically displays the preset parameter and result information after variable filling, and can prompt the bioinformatics commands that are actually applied.

[0097] For the linear process module and the network structure module, the task submission subsystem simultaneously displays the preset parameters of each submodule after variable filling.

[0098] In addition, the task submission subsystem is built on the user system and file system. Specifically, a specific directory in the user's independent space is first set as the project directory, and a project folder is generated in the same directory to store relevant information.

[0099] In one embodiment, the task submission subsystem allows specified steps or nodes to run before the task is submitted, without requiring the module content to be re-edited.

[0100] In this embodiment, for the linear process module and the network structure module, the task submission subsystem can temporarily shut down some running steps or nodes before task submission.

[0101] In one embodiment, the job submission process of the task submission subsystem consists of three steps: generating a task, generating a script, and submitting the job.

[0102] (1) Generate task: After completing the input form, click Generate task to generate a task in the project and generate a corresponding task log folder. The task records all input parameters, which can be viewed, modified and resubmitted later.

[0103] (2) Generate script: Generate the Slurm script to be submitted based on the input variables and the selected template information, and determine whether it is a multi-sample input based on whether the input variables contain wildcards, generate Slurm scripts for all samples, and allow users to manually edit the script content after confirming the script information again.

[0104] In one embodiment, such as Figure 1 As shown in B, the job script is based on the Slurm specification and adopts a three-level nested structure, including a main job layer, a sub-job layer, and the smallest job unit;

[0105] The main job layer is used to submit subjobs, with one subjob corresponding to one sample;

[0106] The sub-job layer is used to submit the smallest job unit;

[0107] The smallest unit of operation includes bioinformatics software commands and scripts, and is used to specify the number of threads to be allocated.

[0108] In one embodiment, the network structure module is used to generate a corresponding number of minimum job units with dependencies based on the number of nodes when each sub-job layer of the command script module and the linear process module submits a minimum job unit.

[0109] In this embodiment, each sub-job of the command script module and the linear process module submits a minimum job unit, and the network structure module generates a corresponding number of minimum job units with dependencies based on the number of nodes.

[0110] In one embodiment, the minimum job unit includes a command script type minimum job unit and a linear process type minimum job unit, wherein the command script type minimum job unit contains a single step, and the linear process type minimum job unit contains multiple sub-steps.

[0111] In this embodiment, statements reporting execution progress are inserted at the start of all jobs and at the start of each step in the smallest job unit within the Slurm script. After a task is submitted, the server executes the included bioinformatics commands according to the script content and saves the execution progress to an intermediate file during the process.

[0112] In the above embodiments, the task submission subsystem generates corresponding Slurm scripts based on the modules and samples. Taking the most complex case as an example, when submitting a network structure module task and inputting variables with wildcards {*}, a master job script is first generated. The master job submits multiple sample sub-jobs; each job corresponds to one sample input, and each sub-job submits multiple minimum job units. There are dependencies between different jobs (sbatch --dependency=afterok); the minimum job unit contains the actual bioinformatics commands or generates the corresponding programming language file based on the "use code".

[0113] If the wildcard {*} is not used, no input sample detection is performed, and the total job submits only one subjob; if there are no dependencies (i.e., non-network structure modules), a subjob submits only one minimum job unit.

[0114] Each level of job generates a corresponding Slurm job ID upon task submission, achieved by adding the parameter "--parsable". The obtained ID is then output ("echo") to an intermediate file. Within the smallest job unit, the time status is input ("echo") to an intermediate file before and after each bioinformatics software command.

[0115] Figure 2 A schematic diagram of the auxiliary system components of the workflow management subsystem in one embodiment is shown, such as... Figure 2 As shown, the implementation of this workflow management subsystem depends on a file system and a database system;

[0116] The file system allocates an empty folder for each user, simulating the / home directory. Through front-end and back-end code, the Linux file space is visualized in a Windows-like manner, providing common operations such as file upload, compression, copy, cut, delete, and rename. It can also display PDFs and images, as well as text.

[0117] The Monaco-editor code editor plugin is embedded in the file system. Developed by Microsoft, this open-source code editor is a core editing component of Visual Studio Code. Designed specifically for web environments, it offers powerful code editing capabilities and a near-desktop IDE experience. Embedding the Monaco Editor plugin in the file system can significantly improve the efficiency of code editing, presentation, and management, making it particularly suitable for online IDEs, code demonstration platforms, and real-time collaborative editing scenarios.

[0118] The file system allows users to manipulate and manage Linux files in a browser. When performing actual bioinformatics analysis tasks, users need to first upload the files to be analyzed to the file system and set a directory as the project directory on the task submission page for subsequent analysis.

[0119] The database system includes the following 7 data tables:

[0120] (1) User information, storing user-related information, such as user account information, preference settings (such as the theme of the text editor Monaco-editor) and the clipboard of the user extraction module;

[0121] (2) Command script module, which stores module system information, various presets, code information and custom comments;

[0122] (3) Linear process module, storing module system information, various presets, sub-modules and custom remarks;

[0123] (4) The network structure module stores system information, various presets, sub-modules, and custom remarks;

[0124] (5) Project list, storing project directories and custom notes, etc.;

[0125] (6) Task list, which stores the project to which it belongs, the module used, the module revisions, and custom notes, etc.;

[0126] (7) Job list, which stores job system information, job running status, job IDs actually submitted to Slurm and running progress, etc.

[0127] Figure 3 A schematic diagram of a workflow management subsystem deployed on a server is shown in one embodiment.

[0128] As shown in the diagram, the core of the workflow management subsystem consists of a file system, a module management system, and a task submission subsystem. All three belong to the www user and are integrated with the database system. The visualization and interactivity of this workflow management subsystem are handled by the front-end code, while information storage and functionality are handled by the back-end code. For administrators and users capable of creating their own modules, accounts are created, and software is installed via SSH. The installed software can then be referenced and used by this workflow management subsystem. For general users or those with only a basic understanding of bioinformatics, modules shared by other users can be used to quickly perform bioinformatics analysis.

[0129] This management system uses HTML, CSS, and JavaScript as its front-end code, and Monaco-editor, D3, Plupload, and jQuery as front-end plugins. Based on this, the file system implements Windows-like file operations, the module management system implements dynamic module dragging and a Directed Acyclic Graph (DAG), and the task submission subsystem implements visual status updates and status management.

[0130] The backend code used in this management system is PHP and MySQL, which, together with the frontend code, enables the process from frontend operations to script generation and submission to the Slurm system.

[0131] Appendix Figure 4 This is a schematic diagram of a command script module embodiment in one example. Taking "trash" as an example, it demonstrates the creation and application of a basic command script module. "trash" is the name of a bioinformatics software mainly used to analyze repetitive sequences in genomic centromeres.

[0132] The software is installed in a user's conda environment. The environment name (default environment name) is "env_centromere", the command name is "TRASH_run.sh", and the command address (default address) is also located in a folder within that user's directory. The default thread count is 10. The above is the basic information of the software, defining its name, address, and environment. At this point, the software can be executed, but since no other parameters have been entered, it will typically output the software version, description, or error message.

[0133] To achieve modularity, visualization, and interactivity, and to improve the flexibility and efficiency of bioinformatics commands, this invention decomposes software parameters into template presets, parameter presets, and variable presets. To make the software easier for users to understand, result presets are also added, which are implemented in the same way as parameter presets but are only used as prompts.

[0134] In this embodiment, the template is preset to contain multiple lines of content, including "{input}". This content is interpreted by the parameter preset (variable passing, i.e., string replacement). The interpreted template is "{input folder} / {name}.fa". If further interpreted by the variable, it becomes ". / ?.fa". The question mark value is converted into an inputtable form on the task submission page to specify the actual content to be input.

[0135] On the task submission page, three input forms—name, input folder, and output folder—are generated based on the preset variables. The default values ​​for the form content are those set in the module. Each variable is assigned a random, highly saturated color, which can be randomly changed by clicking the color box. Regardless of whether the form content or the color is changed, the corresponding values ​​in the parameter presets and result presets will change dynamically at the same time.

[0136] Temporary revisions to template presets and parameter presets are possible, taking effect only upon task submission without altering the content of the corresponding modules.

[0137] Appendix Figure 5 This is a schematic diagram of a linear process module embodiment in one example.

[0138] Main content and appendix Figure 4 The implementation of the command script module is similar, but sub-modules are added below. Each sub-module is a command script module, and temporary revisions of template presets and parameter presets can be made to each sub-module.

[0139] You can specify the execution of certain steps in a linear process module by changing the starting point.

[0140] Appendix Figure 6 This is a schematic diagram of a network structure module embodiment in one example.

[0141] Main content and appendix Figure 4 The implementation of the command script module is similar, but sub-modules are added below. Each sub-module is a command script module or a linear process module. Temporary revisions of template presets and parameter presets can be made to each command script module in the sub-module.

[0142] By controlling the open / closed state of nodes, specific steps of the network structure module can be specified for execution.

[0143] This invention relates to a Slurm-based bioinformatics workflow management system. In terms of bioinformatics command storage, the system provides unified and standardized storage management for various commands, storing information such as the command's usage environment, address, and template parameters in a database for easy and accurate user retrieval, reducing learning and debugging costs. It also provides a comment function for each command and its parameters, allowing users to annotate parameter meanings, applicable scenarios, suggested values, and interaction relationships as needed. This enables users to quickly understand how to use existing commands without needing to consult additional tutorials or help documentation. Furthermore, the system adopts a modular design, breaking down complex analysis tasks into independent modules, allowing users to flexibly combine and build personalized workflows, and supporting the combination and reuse of different modules. In addition, the system establishes a dedicated sharing platform to promote the exchange and sharing of high-quality modules and workflows among users, driving industry development.

[0144] In terms of bioinformatics analysis processes, this system is highly flexible and adaptable to diverse analytical needs. Users can flexibly adjust parameters, module combinations, and task scheduling strategies. It provides a visual operation and monitoring interface, allowing users to intuitively build and edit workflows and monitor task progress and intermediate results in real time. It also automates processes, automatically completing tasks such as environment configuration, script generation, task submission, and progress monitoring. Through these innovative designs, this invention effectively lowers the technical barriers to bioinformatics analysis and improves analytical efficiency and result quality.

[0145] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0146] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A bioinformatics workflow management system based on Slurm, characterized in that, include: Module management subsystem and task submission subsystem; The module management subsystem is used to generate, combine, and manage various modules. Each module in the module management subsystem includes a command script module, a linear process module, and a network structure module. The command script module is used to store bioinformatics software commands and scripts; The linear process module is composed of the command script modules connected in sequence according to the execution order; The network structure module is composed of each of the command script modules or the linear process modules connected through dependencies; The task submission subsystem is used to convert the selected module into a visual task submission page, automatically determine whether there are multiple parallel inputs based on the content of the form input, and further generate and execute a job script that conforms to the Slurm specification.

2. The system according to claim 1, characterized in that, Each module generated by the module management subsystem includes template preset information, parameter preset information, variable preset information, and result preset information; among which... The template preset information records the command line parameters of bioinformatics software commands; The parameter preset information record explains the template preset information. The preset result information is used to indicate the file to be generated; The preset variable information is used to indicate the parameters to be passed in when configuring the actual task.

3. The system according to claim 2, characterized in that, The command script module includes environment preset information, address preset information, thread preset information, and code preset information; wherein... The preset environmental information records the system environment when bioinformatics software commands are used; The preset address information records the command address of the bioinformatics software; The thread preset information records the threads to be allocated by bioinformatics software commands; The code is preset to a custom script.

4. The system according to claim 2, characterized in that, The linear process module also includes thread preset information; The thread preset information records the threads to be allocated by the bioinformatics software commands.

5. The system according to claim 2, characterized in that, The linear process module or network structure module further includes sub-modules, wherein the sub-module of the linear process module is a command script module, and the sub-module of the network structure module is a command script module or a linear process module. The submodule adjusts its template preset information and parameter preset information without changing its content through template revision and parameter revision.

6. The system according to claim 1, characterized in that, The task submission subsystem is also used for: An input form is generated based on preset module variables. The form is used to receive variables input by the user. The input variables include input files and input parameters. The system dynamically displays the preset parameter information and preset result information after the variables are filled in, so as to prompt the bioinformatics commands that are actually applied. In response to the input variables, the preset parameter information of the linear process module and the network structure module after variable filling is dynamically displayed. Upon receiving revision information, the command script module can perform temporary revisions of template presets and parameter presets when acting as a main module or a sub-module, thereby fine-tuning the actual template presets and parameter presets without changing the content of the sub-module.

7. The system according to claim 6, characterized in that, The task submission subsystem allows specified steps or nodes to run before the task is submitted, without requiring the module content to be re-edited.

8. The system according to claim 1, characterized in that, The job script is based on the Slurm specification and adopts a three-level nested structure, including a main job layer, a sub-job layer, and the smallest job unit; The main job layer is used to submit subjobs, with one subjob corresponding to one sample; The sub-job layer is used to submit the smallest job unit; The smallest unit of operation includes bioinformatics software commands and scripts, and is used to specify the number of threads to be allocated.

9. The system according to claim 8, characterized in that, The network structure module is used to generate a corresponding number of minimum job units with dependencies based on the number of nodes when each sub-job layer of the command script module and the linear process module submits a minimum job unit.

10. The system according to claim 8, characterized in that, The minimum job unit includes a command script type minimum job unit and a linear process type minimum job unit, wherein the command script type minimum job unit contains a single step, and the linear process type minimum job unit contains multiple sub-steps.

Images