A cross-platform workflow development and invocation system based on modular management

Abstract

This invention discloses a cross-platform workflow development and invocation system based on modular management, relating to the field of workflow development technology. The system includes: an acquisition unit for acquiring files to be parsed, module files, and command parameters; a parsing unit for obtaining execution instructions; a cloud repository unit for storing cloud-based workflow templates; a local repository unit for storing local workflow templates; and an invocation unit for inputting execution instructions into a transmission tool. The transmission tool obtains a workflow identifier based on the execution instructions, and then obtains a cloud-based workflow template and / or a local workflow template based on the workflow identifier to obtain the target workflow template. This system can solve the problems of steep learning curves, complex configuration and management operations, high learning costs, and susceptibility to environment configuration and development errors, which reduce user experience in existing workflow development and invocation systems.

Description

Technical Field

[0001] This invention relates to the field of workflow development technology, and more specifically, to a cross-platform workflow development and invocation system based on modular management. Background Technology

[0002] Workflows provide a method for automating tasks in a computing environment. Over time, software programming has evolved from writing applications in low-level languages ​​such as assembly to high-level languages ​​like C++, C#, and Visual Basic, such as Nextflow and Snakemake. Nextflow is a Groovy-based workflow management tool that supports complex parallel and distributed computing and has powerful container integration capabilities (supporting Docker, Singularity, etc.). Snakemake is a workflow engine that provides a readable Python-based workflow definition language and a powerful execution environment that can scale from a single-core workstation to a computing cluster without modifying the workflow. Higher-level languages ​​typically remove low-level memory management and other machine-based limitations, allowing programmers to think more from the perspective of the objects representing the problem being solved.

[0003] However, Nextflow uses the Groovy language, which is not user-friendly for general users, resulting in a steep learning curve. Users need to spend a considerable amount of time mastering the language and configuring workflows. Nextflow's configuration file management and output file abstraction make debugging and result management complex, requiring users to have a deep understanding of its internal workings, which is not user-friendly for non-technical users. Snakemake uses the Python language, and its learning curve is also relatively steep, especially when the workflow structure is complex. Users need a deep understanding of Snakemake's rule definitions, DAG construction, and file dependencies.

[0004] Therefore, how to provide a workflow development and invocation system that can be used by non-programming experts and users is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the issues of steep learning curves, complex configuration and management operations, high learning costs, and susceptibility to environment configuration and development errors that reduce user experience associated with existing workflow development and invocation systems, this invention provides a cross-platform workflow development and invocation system based on modular management. The system includes: an acquisition unit for acquiring the file to be parsed, module files, and command parameters;

[0006] Parsing unit: used to obtain execution instructions;

[0007] Cloud Warehouse Unit: Used for storing cloud workflow templates in a cloud-based warehouse;

[0008] Local repository unit: Used to store local workflow templates based on the local repository;

[0009] Invocation unit: used to input the running instruction into the transmission tool, the transmission tool obtains the workflow identifier based on the running instruction, obtains the cloud workflow template and / or the local workflow template based on the workflow identifier, and obtains the target workflow template;

[0010] The execution instructions are obtained as follows: based on predefined parsing tags, the compiler obtains the parsing parameters of the file to be parsed, replaces the predefined parameters with the parsing parameters and the command parameters to obtain execution parameters and execution environment data; based on the environment tags of the execution environment data, the environment type is obtained, and based on the execution parameters and the environment type, the execution instructions are obtained.

[0011] This invention parses user-input files and command parameters, replacing the original parameters without relying on file output. This allows for greater flexibility in adapting to different workflow scenarios. Through a flexible parameter passing mechanism, each step in the workflow can be quickly defined and adjusted without requiring large-scale modifications to the entire process, simplifying workflow definition and management, making development simpler, and resolving issues prone to development errors. The compiler parses input files into unified encoded parameters, eliminating the need for users to learn new high-level programming languages. Users can create and manage complex workflows based on familiar programming syntax, lowering the technical barrier and addressing steep learning curves and high learning costs. Operationally, it is more user-friendly and quicker for non-programming experts and users, improving the user experience. The compiler pulls, starts, and configures the required environment and workflow templates, eliminating the need for user configuration and resolving issues of complex and error-prone environment configuration. This improves the user experience, increases development efficiency, and reduces the time cost of environment debugging.

[0012] Further, the specific steps for obtaining the running instruction based on the running parameters and the environment type include: if the environment type is a process-oriented type, then a plurality of first files are obtained based on the running parameters, first script data is obtained from each of the first files, and the first script data is concatenated based on the calling order of the first files to obtain the running instruction; if the environment type is an isolated type, then second script data for the isolated environment is generated, the running parameters are compiled into third script data for the isolated environment, and the running instruction is obtained based on the second script data and the third script data; if the environment type is a normalized type, then the running instruction is obtained based on the running parameters.

[0013] Furthermore, the parsing unit is also used to obtain a development workflow template. The development workflow template is obtained as follows: the compiler obtains several second files of the module file, obtains fourth script data of each second file based on predefined module parameters, concatenates the fourth script data based on the calling order of the module files to obtain development instructions, and obtains the development workflow template based on the development instructions.

[0014] Furthermore, the calling unit is also used to upload the development workflow template to the cloud repository and / or the local repository.

[0015] It provides a unified management platform through which users can download, install, and share different workflow templates, greatly improving the reusability of workflow templates and enabling users to easily use existing workflow tools, reducing the workload of repetitive configuration and development.

[0016] Furthermore, each cloud-based workflow template and each local workflow template corresponds to a template data set, which includes a unique identifier, upload time, upload user, distinguishing parameters, upload development parameters, and upload environment parameters; each development workflow template corresponds to a development data set, which includes development time, development user, development instructions, and development environment parameters.

[0017] Furthermore, the system also includes a differentiation unit, which is used to obtain the differentiation parameters. The differentiation parameters are obtained by: obtaining environment differentiation data based on the upload environment parameters, the environment differentiation data including operating system, application services, programming language, framework, and environment dependencies; obtaining encoding differentiation data based on the upload development parameters, the encoding differentiation data including the number of files, file calling order, and SQL statements; and obtaining the differentiation parameters based on the environment differentiation data and the encoding differentiation data.

[0018] Differentiating all the templates in the repository can be used to quickly retrieve the target template that the user wants.

[0019] Furthermore, the system also includes a prediction unit, which is used to obtain a predicted workflow template. The predicted workflow template is obtained by: acquiring the user's historical operation data; obtaining the work category, template usage frequency, and template usage time based on the historical operation data; obtaining a first workflow template list based on the work category; acquiring the real-time operation time; obtaining a time range based on the real-time operation time; obtaining a second workflow template list based on the time range, the template usage time, and the first workflow template list; obtaining a third workflow template list based on the template usage frequency and the second workflow template list; and obtaining the predicted workflow template based on a preset list range and the third workflow template list.

[0020] By analyzing the frequency of template usage within historical usage periods, we can infer the templates that users might use in the current time period, reducing the time users spend writing input files and obtaining the target template more quickly.

[0021] Furthermore, the system also includes a similarity unit, which is used to obtain workflow template groups, obtain target similarity based on the workflow template groups, the running parameters and the running environment data, obtain the optimal workflow template based on the target similarity, and update the predicted workflow template to the optimal workflow template;

[0022] The workflow template grouping is obtained as follows: a first similarity is obtained based on the environment difference data; a second similarity is obtained based on the number of files and the order in which the files are called; a third similarity is obtained based on the SQL statement; a total similarity is obtained based on the first similarity, the second similarity, and the third similarity; and the cloud workflow template and the local workflow template are divided based on the total similarity to obtain the workflow template grouping.

[0023] In scenarios with a large number of templates, it is difficult to quickly obtain the target template. By comparing data information in the repository, similar templates are divided into the same group. Based on the similarity between the user-input file and the files in the repository, similar groups can be found more quickly, and the target template can be found in the similar groups.

[0024] Furthermore, the similarity unit is also used to: obtain a first development similarity based on the environment difference data and the development environment parameters; obtain a second development similarity based on the development instructions and the encoding difference data; obtain similar workflow templates based on the first development similarity, the second development similarity, and the workflow template grouping; obtain similar environment parameters and similar development instructions of the similar workflow templates; obtain user modification data; and obtain a new workflow template based on the user modification data, the similar environment parameters, and the similar development instructions.

[0025] In some scenarios, users only need to modify some parameters to obtain a new template. Based on the development instructions input by the user, similar workflow templates can be obtained. By modifying some parameters of the similar workflow templates, a new workflow template can be obtained. By modifying the parameters of existing similar templates, new templates can be created more quickly.

[0026] Furthermore, the system also includes an identical unit, which is used for:

[0027] Determine whether the total similarity is greater than the first preset similarity. If so, obtain several first identical workflow templates based on the total similarity, obtain the same unique identifier of all the first identical workflow templates, and transmit the same unique identifier to the backend for processing.

[0028] Based on the first development similarity and the second development similarity, a third development similarity is obtained. It is then determined whether the third development similarity is greater than the second preset similarity. If so, several second identical workflow templates are obtained based on the third development similarity, and all second identical workflow templates are returned to the user for processing.

[0029] Considering the large number of templates and the development scenarios by developers, there may be identical templates, or templates that already exist in the template repository under development. By determining whether the above situations exist through similarity, we can reduce the waste of resources and reduce the development time for users.

[0030] One or more technical solutions provided by this invention have at least the following technical effects or advantages:

[0031] 1. This invention parses user-input files and command parameters, and the compiler automatically replaces the original parameters without manual replacement or reliance on file output. This allows for greater flexibility in adapting to different workflow scenarios. Through a flexible parameter passing mechanism, each step in the workflow can be quickly defined and adjusted without large-scale modifications to the entire process, simplifying workflow definition and management. The compiler parses input files into unified encoded parameters, eliminating the need for users to learn new high-level programming languages. Users can create and manage complex workflows based on familiar programming syntax, lowering the technical barrier and addressing the steep learning curve issue. This allows non-programming experts and users to quickly get started, improving the user experience. Furthermore, the compiler pulls, starts, and configures the required environment and workflow templates, eliminating the need for user configuration and resolving the complexity and error-prone nature of manual configuration. This improves the user experience, increases development efficiency, and reduces the time cost required for environment debugging.

[0032] 2. Based on the preset list range and the third workflow template list, predictive workflow templates are obtained. By analyzing the usage frequency of templates within historical usage periods, the templates that users may use in the current time period are inferred, reducing the time users spend writing input files and enabling faster acquisition of target templates.

[0033] 3. Based on user-modified data, similar environment parameters, and similar development instructions, a new workflow template is obtained. According to the development instructions input by the user, similar workflow templates are obtained. By modifying some parameters of the similar workflow templates, a new workflow template is obtained. Users only need to modify the parameters of the existing similar templates to obtain new templates, which can create new templates more quickly. Attached Figure Description

[0034] The accompanying drawings, which are provided to further illustrate embodiments of the invention and constitute a part of this invention, are not intended to limit the scope of the invention.

[0035] Figure 1 This is a flowchart illustrating a cross-platform workflow development and invocation system based on modular management, as described in this invention.

[0036] Figure 2 This is a schematic diagram of the structure of a cross-platform workflow development and invocation system based on modular management in this invention. Detailed Implementation

[0037] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, where there is no conflict, the embodiments of the present invention and the features thereof can be combined with each other.

[0038] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0039] Example 1

[0040] refer to Figures 1-2 This embodiment provides a cross-platform workflow development and invocation system based on modular management. The system includes:

[0041] Acquisition Unit: Used to acquire the file to be parsed, module file, and command parameters;

[0042] Parsing unit: used to obtain execution instructions;

[0043] Cloud Warehouse Unit: Used for storing cloud workflow templates in a cloud-based warehouse;

[0044] Local repository unit: Used to store local workflow templates based on the local repository;

[0045] Calling unit: used to input the running instruction into the transmission tool, the transmission tool obtains the workflow identifier based on the running instruction, obtains the cloud workflow template and / or the local workflow template based on the workflow identifier, and obtains the target workflow template; in this embodiment, the workflow identifier can be a unique identifier, index, number or name of the workflow template, or other identifier that can identify the workflow template.

[0046] The method for obtaining the execution instruction is as follows:

[0047] Based on predefined parsing tags, the compiler obtains the parsing parameters of the file to be parsed, replaces the predefined parameters with the parsing parameters and the command parameters to obtain running parameters and running environment data; obtains the environment type based on the environment tags of the running environment data, and obtains the running instructions based on the running parameters and the environment type.

[0048] Based on <workdir>The tag retrieves the path of the file to be parsed, based on <run>The tag retrieves the command line and parsed parameters. Based on command parameters such as IN a (input parameter a) and OUT b (output parameter b), predefined parameters are replaced. For example, if the predefined parameter IN c is used, c is replaced with a, thus obtaining the runtime parameters and runtime environment data. The environment type is obtained based on the environment tag, such as based on... <flow>The label indicates that it is a process-oriented type.<container / apptainer / singularity / podman / docker:…> The label indicates that it is an isolation type. <local>The label determines whether it is a normal type, and then the running instructions are obtained based on the running parameters and environment type.

[0049] The specific steps for obtaining the running instructions based on the running parameters and the environment type include:

[0050] If the environment type is a process-oriented type, then several first files are obtained based on the running parameters, and the first script data in each first file is obtained. Based on the calling order of the first files, the first script data is concatenated to obtain the running instructions. If the file to be parsed contains multiple files, then the first file is obtained through parameters such as file name or file path. Referring to the above method of obtaining the running instructions, the script data in each first file is obtained, and they are concatenated according to the calling order of the files. For example, RUN:PRE, RUN, and RUN:AFTER represent the first call, the second call, and the third call, respectively, thereby obtaining the running instructions.

[0051] If the environment type is an isolation type, then second script data for the isolation environment is generated, the running parameters are compiled into third script data for the isolation environment, and the running instructions are obtained based on the second script data and the third script data; as according to<docker:container-name:image-name:…> In terms of setting the runtime environment, the second script data of the Docker container is automatically generated, thereby realizing the automatic construction of the environment. Using heredoc (a method of defining multi-line strings in PHP), the runtime parameters to be run are passed to the docker exec container… command, and the runtime parameters are recompiled into a third script data that can run inside the Docker container.

[0052] If the environment type is a normalized type, then the running instructions are obtained based on the running parameters.

[0053] A process type refers to a file containing the content or code of other files, i.e., file inclusion.

[0054] Isolation refers to the mutual isolation between containers and between containers and the host machine in terms of resource usage, file system, network, etc., to ensure that containers run in an independent virtual environment, thereby ensuring the security and stability of container operation, such as Docker containers.

[0055] The generalized type refers to other environment types besides the process type and the isolated type.

[0056] In this embodiment, both cloud warehouses and local warehouses can be further divided into two types: container type and workflow category.

[0057] In this embodiment, the transmission tool, the compiler, the cloud repository, and the local repository can be written in programming languages ​​such as Python, Java, C, and C#. The transmission tool is used to transmit files; the compiler is used to parse files; and the cloud repository and the local repository are used to store files.

[0058] In this embodiment, the transmission tool, the compiler, the cloud repository, and the local repository are preferably written in Common Lisp, especially in SBCL (Steel Bank Common Lisp).

[0059] Common Lisp is one of the many dialects of Lisp, a modern, multi-paradigm, high-performance, and compilable standardized ANSI programming language. SBCL is a high-performance implementation of Common Lisp, often referred to as the Ferrari 12 of Common Lisp implementations. SBCL not only provides a complete Lisp programming environment but also boasts a powerful compiler and runtime system.

[0060] Using Common Lisp has the following advantages:

[0061] 1) Common Lisp's macro system is more powerful and flexible than almost any other programming language, allowing the creation of DSLs (Domain-Specific Languages) or other high-level features within compilers and transport tools. This enables the easy embedding of complex workflows and configurations into code without introducing new interpretation or compilation layers. Macros can generate and transform code during the compilation phase, thus automating the generation of complex code logic and reducing the need for manually written, verbose code.

[0062] 2) Common Lisp encourages modular and reusable code structures and provides excellent support for functional programming. Lisp's flexible syntax and functional programming style allow for the integration of core compiler and transport tool functionalities with modular design, enabling users to flexibly combine and extend functionality. This is particularly advantageous for workflow management systems, as new features or modules can be added without modifying the underlying code.

[0063] 3) Common Lisp has a built-in garbage collection mechanism, providing convenience and stability for memory management. For workflow management systems that need to run for a long time and may process large amounts of data, the stability of garbage collection can avoid many memory leak problems, which is very important for ensuring the robustness of the system;

[0064] 4) Common Lisp runs faster than languages ​​like Python, while having higher abstraction capabilities than languages ​​like C / C++, thus maintaining high execution speed while possessing a high level of abstraction.

[0065] 5) Common Lisp has a situation handling system that other languages ​​do not have, which can handle various situations and errors more robustly and interactively. This is a very important and advantageous advantage in DSL.

[0066] In this embodiment, the file to be parsed and the module file can be written in programming languages ​​such as Python, Java, C, and C#, preferably in TAF language, and parsed into shell language by a compiler.

[0067] A TAF file is the source code file for TAF, used to store TAF code in plain text form. The file contains function, class, variable definitions and execution logic, and is the main place for writing programs.

[0068] TAF is a Domain Specific Language (DSL), specifically designed for terminal command-line operations (Shell) to facilitate containerized workflow design and management. TAF is case-sensitive, indentation-insensitive, uses only newlines as separators, and comments begin with #. The syntax follows a header-based system, primarily consisting of three levels of tags from top to bottom:

[0069] 1) Top-level identifier: Located on the first line of each file and unique, used to label the type (tool or workflow) of the .taf file and give it a name, such as +TOOL:xxx / +FLOW:xxx;

[0070] 2) First-level identifiers: such as LOAD / ARGS / RUN (RUN:PRE / RUN / RUN:AFTER), etc. LOAD is used to call the TAF file, ARGS is used to set parameters, and RUN is used to set the running content. There can only be one first-level identifier in each file. If there are two, the first one takes precedence and all subsequent identifiers with the same name are ignored.

[0071] 3) Secondary identifier: <xxx>Enclosed in angle brackets, the specific content and purpose depend on the first-level identifier; different first-level identifiers have different content and usages. For example, the second-level identifier under LOAD is used to specify the path or alias of the TAF file, and can be represented as:

[0072] LOAD

[0073]

[0074] . / tools / a.taf;

[0075] The second-level identifier under ARGS is used to define the value of a variable, and parameters can span multiple lines under the identifier. Variable substitution is done using ::xxx:: to call parameters, and calls can be made across files. For example, a .taf file can use LOAD to call a.taf file, and... As an alias for a.taf, you can use ::a:xxx:: to access the xxx variable in a.taf and perform variable substitution before runtime;

[0076] It also provides built-in environment variables to help developers quickly build TAF workflow templates. For example, `::*WORKDIR*::` represents the current working directory; `::*LOAD-DIR*::` represents the path to the TAF file being run; `::*CPUS*::` represents the number of CPU threads on the current operating platform; `::*MAIN*::` is a variable for the built-in `if` command used for quick tool building. If the user does not pass any parameters, then `if = ::else::`. If parameters are passed and begin with `--xxx`, the official build method is used, e.g., `if = ::main::`. If it does not begin with `--xxx`, then the user-passed new command is passed directly. If there is a TAF file:

[0077] +TOOL:blast

[0078] ARGS

[0079] <mian>

[0080] ::cmd::::opts::::in::

[0081] <else>

[0082] cat::*LOAD-DIR*::. / blast.info .txt

[0083] RUN

[0084] <container:taf-blast: docker.io / ncbi / blast:latest >

[0085] ::*MAIN*::

[0086] As shown in the above taf file, users can use it in the following three ways:

[0087] (1) taf-blast: If the user does not pass any parameters, it calls ::else::, which displays help;

[0088] (2) taf-blast—cmd blastn—in xxx—opts yyy: User-input parameters, starting with —cmd, run in the ::main:: manner;

[0089] (3) taf-blast blastn xxx yyy…: User-input parameters, if not starting with -xxx, directly pass blastnxxx yyy… to the position of ::*MAIN*::.

[0090] RUN is the code section of a .taf file that contains all the commands to be executed. RUN can be divided into three flags: RUN:PRE, RUN, and RUN:AFTER, and the contents of RUN will be executed in that order. There are five main secondary flags for RUN:

[0091] 1) <local sh shell>: Corresponds to local shell code. Shell code can be written under this identifier. This part of the shell code will be directly copied over without modification during the conversion.

[0092] 2)<container / apptainer / singularity / docker / podman:container-name:image-name(@...$...)> This corresponds to code running inside containers such as apptainers, docker, or podman. When this tag is detected, it will automatically detect available containers and images locally. If multiple containers and images exist locally, the installation order can be specified in the configuration file, such as the congif configuration file, or specify the first detected container or image to install. If the corresponding image and container do not exist locally, the compiler will automatically generate code to obtain the image and generate the container to automatically configure the environment. If they exist, the corresponding container will be used directly.

[0093] 3)<python / R / …> : Write code in the corresponding language, pass it to the corresponding compiler, and then output the running result;

[0094] 4)<sh:…> This identifier allows the corresponding command following sh to be used as the shell command header, and then the code below sh can be passed to the command via heredoc;

[0095] 5) <flow>The core identifier of the workflow automatically recognizes the taf commands within it, pre-runs and replaces them with shell commands, and then constructs the flow in the form of written shell scripts.

[0096] In this embodiment, the parsing unit is further configured to: run the first script data based on the pre-running parameters to obtain output parameters, and concatenate the output parameters based on the first file calling order to obtain the running instructions;

[0097] In any location within a .taf file, you can use the pre-run parameter $$command$$ to pre-run script data in the shell and use its output as the corresponding value for that script data. For example, you can use RUN- <local>Using $$taf-a–dry-run…$$, $$taf-b–dry-run…$$ to pre-call different taf files and embed their converted shell code into the current taf file, it is possible to build processes quickly like writing shell scripts while ensuring that there are no issues such as environment conflicts.

[0098] In a more preferred embodiment, under RUN <flow>The tags have been optimized to automatically recognize the commands "taf-xxx" and "taffish xxx.taf" and replace them with the corresponding shell text after pre-running, eliminating the need to manually add $$...$$; and it provides <auto-flow>Tags, in <flow>Based on this, it will automatically identify and install any taf-xxx that appears, and pre-run "taf install-n xxx", which means it will automatically identify and install dependencies.

[0099] Using the TAF language, which is almost identical to Shell syntax, users no longer need to learn complex programming languages ​​or master new technologies. For users, familiarity with Shell commands allows for quick workflow construction, shortening workflow development time. Furthermore, the original shell script can be converted to a TAF script in just two steps with minimal cost: (1) Add a TAF header to the beginning of the original script and use... <flow>or <auto-flow>Tag; (2) Add taf-xxx before all taf tools for environment adaptation. Especially for bioinformaticians and researchers without a programming background, the learning curve is greatly reduced compared to Nextflow's Groovy language or Snakemake's complex rule definitions.

[0100] In this embodiment, the parsing unit is further configured to: convert the development workflow template into an executable file, and obtain the development workflow template based on the executable file. For example, PyInstaller is a popular Python code packaging tool that can package Python scripts into executable files.

[0101] You can directly install the module using `taf install xxx`, and then use `taf-xxx` to call the module. You can also use `taf-xxx-h` to view the module's help and usage instructions. This is more in line with shell scripting and reduces the learning curve.

[0102] In this embodiment, the system can run on different operating systems or hardware platforms, i.e., it can run across platforms.

[0103] Example 2

[0104] refer to < / flow> < / flow> < / flow> < / local> < / flow> < / local> < / else> < / mian> Figures 1-2 Based on Embodiment 1, in this embodiment, the parsing unit is further used to obtain a development workflow template, and the development workflow template is obtained in the following way:

[0105] The compiler obtains several second files from the module file, obtains fourth script data for each second file based on predefined module parameters, concatenates the fourth script data based on the calling order of the module files to obtain development instructions, and obtains the development workflow template based on the development instructions.

[0106] The calling unit is also used to upload the development workflow template to the cloud repository and / or the local repository.

[0107] If the module file includes four files: m, n, q, and p, then based on <flow>The compiler parses the four files, referring to the method for obtaining execution instructions, to obtain script data. It then assembles the script data according to the calling order to obtain development instructions and encapsulates them into a module package, thereby obtaining the development workflow template.

[0108] Example 3

[0109] Based on the above embodiments, in this embodiment, each cloud workflow template and each local workflow template corresponds to a template data, which includes a unique identifier, upload time, upload user, distinguishing parameters, upload development parameters, and upload environment parameters;

[0110] Each of the aforementioned development workflow templates corresponds to a set of development data, which includes development time, development user, development instructions, and development environment parameters.

[0111] The system further includes a differentiation unit, which is used to obtain the differentiation parameters. The differentiation parameters are obtained in the following manner:

[0112] Environment differentiation data is obtained based on the uploaded environment parameters. The environment differentiation data includes operating system, application services, programming language, frameworks and programming language required by the programming language, database and environment dependencies required by the application services.

[0113] Based on the uploaded development parameters, encoding difference data is obtained, which includes the number of files, the file access order, and the SQL statement.

[0114] The difference parameters are obtained based on the environmental difference data and the encoding difference data.

[0115] In this embodiment, the application services may include databases, application servers, message queues, caching services, and log monitoring tools, etc.

[0116] The system further includes a prediction unit, which is used to obtain a prediction workflow template. The prediction workflow template is obtained in the following way:

[0117] Obtain the user's historical operation data, and based on the historical operation data, obtain the work category, template usage frequency, and template usage time; and based on the work category, obtain a first workflow template list.

[0118] Obtain the real-time operation time, obtain a time range based on the real-time operation time, and obtain a second workflow template list based on the time range, the template usage time, and the first workflow template list;

[0119] Based on the usage frequency of the templates and the second workflow template list, a third workflow template list is obtained;

[0120] The predicted workflow template is obtained based on the preset list range and the third workflow template list.

[0121] For example, if the current time is 10:00, first obtain the user's work category, then obtain the first workflow template list used by the user based on the work category, then obtain the workflow templates used by the user in the time period of 9:00-11:00 AM within a month to obtain the second workflow template list, then obtain the usage frequency of each template in the second workflow template list, and select the top 2 workflow templates with the highest usage frequency to obtain the predicted workflow templates.

[0122] Example 4

[0123] Based on the above embodiments, in this embodiment, the system further includes a similarity unit, which is used to obtain workflow template groups, obtain target similarity based on the workflow template groups, the running parameters and the running environment data, obtain the optimal workflow template based on the target similarity, update the predicted workflow template to the optimal workflow template, and compare it with the workflow template groups according to the running parameters and running environment data to obtain the template with the highest similarity and obtain the optimal workflow template.

[0124] The workflow template grouping is obtained as follows:

[0125] Based on the environmental difference data, a first similarity is obtained; based on the number of files and the order in which the files are called, a second similarity is obtained; based on the SQL statement, a third similarity is obtained; based on the first similarity, the second similarity, and the third similarity, a total similarity is obtained; based on the total similarity, the cloud workflow template and the local workflow template are divided to obtain the workflow template grouping.

[0126] To obtain the first similarity, we can calculate the similarity based on the number of identical operating systems, application services, programming languages, frameworks, and environment dependencies: First similarity = Number of identical dependencies / Total number of dependencies. We can also obtain file similarity by obtaining the number of files and the order in which files are accessed, assigning different weights to each and summing them to obtain the second similarity. Finally, we can obtain the similarity of identifiers, their number, and their order in SQL statements, assigning different weights to each and summing them to obtain the third similarity. By assigning different weights to the first, second, and third similarities, we can obtain the total similarity. The total similarity can then be divided into different ranges to obtain workflow template groups.

[0127] Alternatively, data can be segmented based on environmental and encoding differences, and then similarity can be calculated. For example, data can be segmented based on operating systems first, and then similarity can be calculated sequentially.

[0128] Example 5

[0129] Based on the above embodiments, in this embodiment, the similar unit is further used for:

[0130] Based on the environmental difference data and the development environment parameters, a first development similarity is obtained;

[0131] Based on the development instructions and the encoding difference data, a second development similarity is obtained;

[0132] Based on the first development similarity, the second development similarity, and the workflow template grouping, similar workflow templates are obtained;

[0133] Referring to the method for calculating similarity in Example 4, the first development similarity and the second development similarity are obtained. They are then compared with workflow template groups to obtain the template with the highest similarity as the similar workflow template.

[0134] Obtain similar environment parameters and similar development instructions for the similar workflow templates; obtain user-modified data;

[0135] Based on the user-modified data, the similar environment parameters, and the similar development instructions, a new workflow template is obtained. The user-modified parameters are then replaced to obtain the new workflow template.

[0136] Example 6

[0137] Based on the above embodiments, in this embodiment, the system further includes an identical unit, which is used for:

[0138] Determine whether the total similarity is greater than the first preset similarity. If so, obtain several first identical workflow templates based on the total similarity, obtain the same unique identifier of all the first identical workflow templates, and transmit the same unique identifier to the backend for processing.

[0139] Based on the first development similarity and the second development similarity, a third development similarity is obtained. It is then determined whether the third development similarity is greater than the second preset similarity. If so, several second identical workflow templates are obtained based on the third development similarity, and all second identical workflow templates are returned to the user for processing.

[0140] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0141] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.< / flow> < / xxx> < / local> < / flow> < / run> < / workdir>

Claims

1. A cross-platform workflow development and invocation system based on modular management, characterized in that, The system includes: Acquisition Unit: Used to acquire the file to be parsed, module file, and command parameters; Parsing unit: used to obtain execution instructions; Cloud Warehouse Unit: Used for storing cloud workflow templates in a cloud-based warehouse; Local repository unit: Used to store local workflow templates based on the local repository; Invocation unit: used to input the running instruction into the transmission tool, the transmission tool obtains the workflow identifier based on the running instruction, obtains the cloud workflow template and / or the local workflow template based on the workflow identifier, and obtains the target workflow template; The method for obtaining the execution instruction is as follows: Based on predefined parsing tags, the compiler obtains the parsing parameters of the file to be parsed, and replaces the predefined parameters based on the parsing parameters and the command parameters to obtain the running parameters and running environment data; The environment type is obtained based on the environment label of the operating environment data, and the operating instruction is obtained based on the operating parameters and the environment type. The specific steps for obtaining the execution instruction based on the execution parameters and the environment type include: If the environment type is a process type, then based on the running parameters, several first files are obtained, first script data in each first file is obtained, and based on the calling order of the first files, the first script data is concatenated to obtain the running instructions; If the environment type is an isolation type, then the second script data of the isolation environment is generated, the running parameters are compiled into the third script data of the isolation environment, and the running instructions are obtained based on the second script data and the third script data; If the environment type is a normalized type, then the running instructions are obtained based on the running parameters; The parsing unit is also used to obtain a development workflow template, and the development workflow template is obtained in the following way: The compiler obtains several second files from the module file, obtains fourth script data for each second file based on predefined module parameters, concatenates the fourth script data according to the calling order of the module files to obtain development instructions, and obtains the development workflow template based on the development instructions. The system further includes a prediction unit, which is used to obtain a prediction workflow template. The prediction workflow template is obtained in the following way: Obtain the user's historical operation data, and based on the historical operation data, obtain the work category, template usage frequency, and template usage time; and based on the work category, obtain a first workflow template list. Obtain the real-time operation time, obtain a time range based on the real-time operation time, and obtain a second workflow template list based on the time range, the template usage time, and the first workflow template list; Based on the usage frequency of the templates and the second workflow template list, a third workflow template list is obtained; The predicted workflow template is obtained based on the preset list range and the third workflow template list.

2. The cross-platform workflow development and invocation system based on modular management according to claim 1, characterized in that, The calling unit is also used to upload the development workflow template to the cloud repository and / or the local repository.

3. The cross-platform workflow development and invocation system based on modular management according to claim 2, characterized in that, Each cloud workflow template and each local workflow template corresponds to a template data, which includes a unique identifier, upload time, upload user, distinguishing parameters, upload development parameters, and upload environment parameters. Each of the aforementioned development workflow templates corresponds to a set of development data, which includes development time, development user, development instructions, and development environment parameters.

4. The cross-platform workflow development and invocation system based on modular management according to claim 3, characterized in that, The system further includes a differentiation unit, which is used to obtain the differentiation parameters. The differentiation parameters are obtained in the following manner: Environment differentiation data is obtained based on the uploaded environment parameters. The environment differentiation data includes operating system, application services, programming language, framework, and environment dependencies. Based on the uploaded development parameters, encoding difference data is obtained, which includes the number of files, the file access order, and the SQL statement. The difference parameters are obtained based on the environmental difference data and the encoding difference data.

5. A cross-platform workflow development and invocation system based on modular management as described in claim 4, characterized in that, The system further includes a similarity unit, which is used to obtain workflow template groups, obtain target similarity based on the workflow template groups, the running parameters and the running environment data, obtain the optimal workflow template based on the target similarity, and update the predicted workflow template to the optimal workflow template. The workflow template grouping is obtained as follows: Based on the environmental difference data, a first similarity is obtained; based on the number of files and the order in which the files are accessed, a second similarity is obtained. A third similarity is obtained based on the SQL statement; a total similarity is obtained based on the first similarity, the second similarity, and the third similarity; the cloud workflow template and the local workflow template are divided based on the total similarity to obtain the workflow template grouping.

6. A cross-platform workflow development and invocation system based on modular management as described in claim 5, characterized in that, The similar unit is also used for: Based on the environmental difference data and the development environment parameters, a first development similarity is obtained; Based on the development instructions and the encoding difference data, a second development similarity is obtained; Based on the first development similarity, the second development similarity, and the workflow template grouping, similar workflow templates are obtained; Obtain similar environment parameters and similar development instructions for the similar workflow templates; Retrieve user-modified data; A new workflow template is obtained based on the user-modified data, the similar environment parameters, and the similar development instructions.

7. A cross-platform workflow development and invocation system based on modular management as described in claim 6, characterized in that, The system also includes an identical unit, which is used for: Determine whether the total similarity is greater than the first preset similarity. If so, obtain several first identical workflow templates based on the total similarity, obtain the same unique identifier of all the first identical workflow templates, and transmit the same unique identifier to the backend for processing. Based on the first development similarity and the second development similarity, a third development similarity is obtained. It is then determined whether the third development similarity is greater than the second preset similarity. If so, several second identical workflow templates are obtained based on the third development similarity, and all second identical workflow templates are returned to the user for processing.

Images