Cross-platform workflow development and calling system based on modular management

Abstract

The invention discloses a cross-platform workflow development and calling system based on modular management, and relates to the technical field of workflow development, and the system comprises an obtaining unit which is used for obtaining a to-be-analyzed file, a module file and a command parameter; the analysis unit is used for obtaining a running instruction; the cloud warehouse unit is used for storing a cloud workflow template based on a cloud warehouse; the local warehouse unit is used for storing a local workflow template based on a local warehouse; the calling unit is used for inputting the operation instruction into a transmission tool, and the transmission tool obtains a workflow identifier based on the operation instruction, obtains a cloud workflow template and / or a local workflow template based on the workflow identifier, and obtains a target workflow template; the problems that due to the fact that an existing workflow development and calling system is steep in learning curve and complex in configuration and management operation, the learning cost is high, environment configuration errors and development errors are prone to occurring, and the user experience feeling is reduced can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of workflow development, and in particular to a cross-platform workflow development and calling system based on modular management. Background Art

[0002] Workflows provide a method for automating tasks in a computing environment. Over time, software programming has evolved from writing applications in low-level languages ​​such as assembly to writing in high-level languages ​​such as C++, C#, and Visual Basic, such as Nextflow and Snakemake. Nextflow is a workflow management tool based on the Groovy language that supports complex parallel and distributed computing and has powerful container integration capabilities (supports Docker, Singularity, etc.); Snakemake is a workflow engine that provides a readable Python-based workflow definition language and a powerful execution environment that can be expanded from a single-core workstation to a computing cluster without modifying the workflow. Higher-level languages ​​typically remove low-level management of memory and other machine-based restrictions, which helps allow programmers to think more in terms of objects that represent the problems to be solved.

[0003] However, Nextflow uses the Groovy language, which is not user-friendly, making its learning curve steep. Users need to spend more time mastering the language and configuring workflows. Nextflow's configuration file management and output file abstraction make debugging and result management more complicated. Users need to have a deep understanding of its internal working mechanism, which is not friendly to non-technical users. Snakemake uses the Python language, and its learning curve is still relatively steep, especially when the workflow structure is complex. Users need to have a deeper understanding of Snakemake's rule definition, DAG construction, and file dependencies.

[0004] Therefore, how to provide a workflow development and calling system for non-programming experts and users is a problem that those skilled in the art need to solve urgently. Summary of the invention

[0005] In order to solve the problem that the existing workflow development and calling system has a steep learning curve, complex configuration and management operations, resulting in high learning costs, prone to environmental configuration errors and development errors, and reduced user experience, the present invention provides a cross-platform workflow development and calling system based on modular management, the system comprising: an acquisition unit: used to acquire a file to be parsed, a module file and a command parameter;

[0006] Parsing unit: used to obtain running instructions;

[0007] Cloud warehouse unit: used to store cloud workflow templates based on cloud warehouse;

[0008] Local warehouse unit: used to store local workflow templates based on the local warehouse;

[0009] A calling unit: used for inputting the running instruction into a transmission tool, wherein the transmission tool obtains a workflow identifier based on the running instruction, obtains the cloud-based workflow template and / or the local workflow template based on the workflow identifier, and obtains a target workflow template;

[0010] The operating instructions are obtained as follows: based on a predefined parsing tag, the compiler obtains the parsing parameters of the file to be parsed, replaces the predefined parameters based on the parsing parameters and the command parameters, and obtains the operating parameters and operating environment data; based on the environment tag of the operating environment data, the environment type is obtained, and the operating instructions are obtained based on the operating parameters and the environment type.

[0011] The present invention parses the files and command parameters input by the user, replaces the original parameters, does not rely on file output, can adapt to different types of workflow scenarios more flexibly, and through a flexible parameter transfer mechanism, quickly defines and adjusts each step in the workflow without large-scale modification of the entire process, simplifies the definition and management of the workflow, makes development simpler, and solves the problem of easy development errors; a compiler is used to parse the input file into a unified encoding parameter, and the user does not need to learn a new high-level programming language, and can create and manage complex workflows based on familiar programming syntax, which lowers the technical threshold and can solve the problems of steep learning curve and high learning cost. In terms of operation, it is more friendly and quick to get started for non-programming experts and users, and improves the user experience; a compiler is used to pull, start and configure the environment and workflow templates required by the user, without the need for the user to configure it by himself, which solves the problem of complex configuration by the user and easy errors in the environment configuration, improves the user experience and development efficiency, and also reduces the time cost required for environment debugging.

[0012] Furthermore, the specific steps of obtaining the running instructions based on the running parameters and the environment type include: if the environment type is a process-based type, obtaining a plurality of first files based on the running parameters, obtaining the first script data in each of the first files, splicing the first script data based on the calling order of the first files, and obtaining the running instructions; if the environment type is an isolated type, generating second script data for an isolated environment, compiling the running parameters into third script data for the isolated environment, and obtaining the running instructions based on the second script data and the third script data; if the environment type is a normalized type, obtaining the running instructions based on the running parameters.

[0013] Furthermore, the parsing unit is also used to obtain a development workflow template, and the method for obtaining the development workflow template is: the compiler obtains several second files of the module file, obtains fourth script data of each second file based on predefined module parameters, splices the fourth script data based on the calling order of the module files, and obtains development instructions; and obtains the development workflow template based on the development instructions.

[0014] Furthermore, the calling unit is also used to upload the development workflow template to the cloud warehouse and / or the local warehouse.

[0015] It provides a unified management platform through which users can download, install and share different workflow templates, greatly improving the reusability of workflow templates, allowing users to easily use existing workflow tools and reducing the workload of repeated configuration and development.

[0016] Furthermore, each of the cloud-based workflow templates and each of the local workflow templates corresponds to a template data, and the template data includes a unique identifier, upload time, upload user, distinguishing parameters, upload development parameters and upload environment parameters; each of the development workflow templates corresponds to a development data, and the development data includes development time, development user, the development instructions and development environment parameters.

[0017] Furthermore, the system also includes a differentiation unit, which is used to obtain the differentiation parameters, and the differentiation parameters are obtained in the following manner: obtaining environment differentiation data based on the uploaded environment parameters, the environment differentiation data including operating system, application service, programming language, framework and environment dependencies; obtaining encoding differentiation data based on the uploaded development parameters, the encoding differentiation data including file quantity, file calling order and SQL statements; obtaining the differentiation parameters based on the environment differentiation data and the encoding differentiation data.

[0018] Differentiating all templates in the warehouse can be used to quickly obtain the target template that the user wants.

[0019] Furthermore, the system also includes a prediction unit, which is used to obtain a predicted workflow template, and the predicted workflow template is obtained in the following manner: obtaining historical operation data of the user, obtaining a work category, a template usage frequency and a template usage time based on the historical operation data, and obtaining a first workflow template list based on the work category; obtaining real-time operation time, obtaining a time range based on the real-time operation time, and obtaining a second workflow template list based on the time range, the template usage time and the first workflow template list; obtaining a third workflow template list based on the template usage frequency and the second workflow template list; and obtaining the predicted workflow template based on a preset list range and the third workflow template list.

[0020] Based on the usage frequency of templates in the historical usage time period, the templates that users may use in the current time period are inferred, which reduces the time users spend writing and importing files and enables them to obtain target templates more quickly.

[0021] Furthermore, the system further comprises a similarity unit, which is used to obtain a workflow template group, obtain a target similarity based on the workflow template group, the operating parameters and the operating environment data, obtain an optimal workflow template based on the target similarity, and update the predicted workflow template to the optimal workflow template;

[0022] The workflow template grouping is obtained by: obtaining a first similarity based on the environment distinction data; obtaining a second similarity based on the number of files and the file calling order; obtaining a third similarity based on the SQL statement; obtaining a total similarity based on the first similarity, the second similarity and the third similarity, and dividing the cloud workflow template and the local workflow template based on the total similarity to obtain the workflow template grouping.

[0023] Considering the scenario of a large number of templates, it is difficult to quickly obtain the target template. By comparing the data information in the warehouse, similar templates can be divided into the same group. According to the similarity between the user's input files and the files in the warehouse, similar groups can be found more quickly, so as to find the target template in the similar groups.

[0024] Furthermore, the similarity unit is also used to: obtain a first development similarity based on the environment distinction data and the development environment parameters; obtain a second development similarity based on the development instructions and the coding distinction data; obtain a similar workflow template based on the first development similarity, the second development similarity and the workflow template grouping; obtain similar environment parameters and similar development instructions of the similar workflow template; obtain user modification data; and obtain a new workflow template based on the user modification data, the similar environment parameters and the similar development instructions.

[0025] Considering some scenarios, the user only needs to modify some parameters to obtain a new template. Then, according to the development instructions input by the user, similar workflow templates are obtained, and some parameters of the similar workflow templates are modified to obtain a new workflow template. By modifying the parameters of the existing similar templates, new templates can be created faster.

[0026] Furthermore, the system further comprises the same unit, and the same unit is used for:

[0027] Determine whether the total similarity is greater than a first preset similarity, and if so, obtain a plurality of first identical workflow templates based on the total similarity, obtain the same unique identifier of all the first identical workflow templates, and transmit the same unique identifier to the backend for processing;

[0028] And based on the first development similarity and the second development similarity, a third development similarity is obtained, and it is determined whether the third development similarity is greater than a second preset similarity. If so, a plurality of second identical workflow templates are obtained based on the third development similarity, and all the second identical workflow templates are returned to the user end for processing.

[0029] Considering a large number of templates and scenarios developed by developers, the same template may exist, or already exist in the template repository being developed. By judging whether the above situation exists through similarity, it can reduce resource waste and reduce user development time.

[0030] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0031] 1. The present invention parses the files and command parameters passed in by the user, and the compiler automatically replaces the original parameters without manual replacement and without relying on file output. It can adapt to different types of workflow scenarios more flexibly, and through a flexible parameter transfer mechanism, each step in the workflow can be quickly defined and adjusted without large-scale modification of the entire process, thereby simplifying the definition and management of the workflow; the compiler is used to parse the incoming files into unified encoding parameters, and the user does not need to learn a new high-level programming language, and can create and manage complex workflows based on familiar programming syntax, thereby lowering the technical threshold and solving the problem of a steep learning curve, so that non-programming experts and users can quickly get started and improve the user experience; the compiler is used to pull, start and configure the environment and workflow templates required by the user, without the need for the user to configure it by himself, thereby solving the problem of complex and error-prone configuration by the user, improving the user experience and development efficiency, and also reducing the time cost required for environment debugging.

[0032] 2. Based on the preset list range and the third workflow template list, obtain the predicted workflow template. Through the usage frequency of the template in the historical usage time period, infer the template that the user may use in the current time period, reduce the time for the user to write the incoming file, and obtain the target template more quickly.

[0033] 3. Based on user-modified data, similar environment parameters and similar development instructions, a new workflow template is obtained. According to the development instructions input by the user, similar workflow templates are obtained, and some parameters of the similar workflow templates are modified to obtain a new workflow template. Users only need to modify the parameters of existing similar templates to obtain a new template, so new templates can be created faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of the present invention, and do not constitute a limitation on the embodiments of the present invention;

[0035] Figure 1 It is a flow chart of a cross-platform workflow development and calling system based on modular management in the present invention;

[0036] Figure 2 It is a structural diagram of a cross-platform workflow development and calling system based on modular management in the present invention. DETAILED DESCRIPTION

[0037] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those within the scope of this description. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0039] Embodiment 1

[0040] refer to Figure 1-Figure 2 This embodiment provides a cross-platform workflow development and calling system based on modular management, the system comprising:

[0041] Acquisition unit: used to obtain files to be parsed, module files and command parameters;

[0042] Parsing unit: used to obtain running instructions;

[0043] Cloud warehouse unit: used to store cloud workflow templates based on cloud warehouse;

[0044] Local warehouse unit: used to store local workflow templates based on the local warehouse;

[0045] Calling unit: used to input the running instruction into the transmission tool, the transmission tool obtains the workflow identifier based on the running instruction, obtains the cloud workflow template and / or the local workflow template based on the workflow identifier, and obtains the target workflow template; in this embodiment, the workflow identifier can be a unique identifier, index, number or name of the workflow template, etc., which can identify the workflow template.

[0046] The operation instruction is obtained as follows:

[0047] Based on the predefined parsing tag, the compiler obtains the parsing parameters of the file to be parsed, replaces the predefined parameters based on the parsing parameters and the command parameters, and obtains the operating parameters and operating environment data; obtains the environment type based on the environment tag of the operating environment data, and obtains the operating instructions based on the operating parameters and the environment type.

[0048] As based on <workdir>The tag obtains the path of the file to be parsed, based on <run>The tag obtains the command line, obtains the parsing parameters, and replaces the predefined parameters based on the command parameters such as IN a (input parameter a) and OUT b (output parameter b). For example, if the predefined parameter IN c is used, c is replaced with a to obtain the running parameters and running environment data; the environment type is obtained based on the environment tag, such as <flow>The tag determines that it is a process type.<container / apptainer / singularity / podman / docker:…> The tag determines that it is an isolated type. <local>The tag determines that it is a normalized type, and then obtains the running instructions based on the running parameters and environment type.

[0049] The specific steps of obtaining the operating instruction based on the operating parameters and the environment type include:

[0050] If the environment type is a process-based type, several first files are obtained based on the running parameters, the first script data in each of the first files is obtained, and the first script data is spliced ​​based on the calling order of the first files to obtain the running instructions; if the file to be parsed contains multiple files, the first file is obtained through parameters such as file name or file path, and the script data in each first file is obtained with reference to the above-mentioned method of obtaining the running instructions, and they are spliced ​​according to the calling order of the files, such as RUN:PRE, RUN and RUN:AFTER respectively represent first call, call and after call, so as to obtain the running instructions.

[0051] If the environment type is an isolated type, second script data of the isolated environment is generated, the operating parameters are compiled into third script data of the isolated environment, and the operating instructions are obtained based on the second script data and the third script data;<docker:container-name:image-name:…> The second script data of the Docker container is automatically generated for the operating environment settings, thereby realizing the automatic construction of the environment. Heredoc (a method of defining multi-line strings in PHP) is used to pass the operating parameters to be run into the docker exec container... command, and the operating parameters are recompiled into the third script data that can be run in the Docker container.

[0052] If the environment type is a normalized type, the operating instruction is obtained based on the operating parameters.

[0053] The process type means that a file contains the content or code of other files, that is, file inclusion.

[0054] Isolation type refers to the isolation between containers and between containers and host machines in terms of resource usage, file system, network, etc., ensuring that containers run in independent virtual environments, thereby ensuring the security and stability of container operation, such as Docker containers.

[0055] Normalized types refer to other environment types except process-based types and isolated types.

[0056] In this embodiment, both the cloud warehouse and the local warehouse can be subdivided into two types: container type and workflow type.

[0057] In this embodiment, the transmission tool, the compiler, the cloud warehouse and the local warehouse can be written in programming languages ​​such as Python, Java, C and C#. The transmission tool is used to transmit files; the compiler is used to parse files, and the cloud warehouse and the local warehouse are used to store files.

[0058] In this embodiment, the transmission tool, the compiler, the cloud warehouse and the local warehouse are preferably written in Common Lisp language, especially SBCL (Steel Bank Common Lisp) of Common Lisp.

[0059] Common Lisp is one of the many dialects of Lisp. It is a modern, multi-paradigm, high-performance and compilable standardized ANSI programming language. SBCL is a high-performance implementation of Common Lisp, known as the Ferrari 12 in Common Lisp implementation. SBCL not only provides a complete Lisp programming environment, but also has a powerful compiler and runtime system.

[0060] Writing in Common Lisp has the following advantages:

[0061] 1) Common Lisp's macro system is more powerful and flexible than almost any other programming language. It can create DSLs (domain-specific languages) or other advanced features in compilers and transport tools, and easily embed complex workflows and configurations into the code without introducing new interpretation or compilation layers. Macros can generate and transform code during the code compilation phase, thereby automatically generating complex code logic and reducing lengthy handwritten code;

[0062] 2) Common Lisp encourages modular, reusable code structures and has good support for functional programming. Through Lisp's flexible syntax and functional programming style, the core functions of compilers and transmission tools can be combined with modular design, allowing users to flexibly combine and expand functions. This is especially beneficial for workflow management systems because new functions or new modules can be added without changing the underlying code.

[0063] 3) Common Lisp has a built-in garbage collection mechanism, which provides convenience and stability for memory management. For workflow management systems that need to run for a long time and may process a large amount of data, the stability of garbage collection can avoid many memory leaks, which is very important for ensuring the robustness of the system;

[0064] 4) Common Lisp runs faster than Python and other languages, and its abstraction ability is higher than C / C++ and other languages. It can maintain high abstraction ability while ensuring running speed.

[0065] 5) Common Lisp has a situation handling system that other languages ​​do not have. It can handle various situations and errors more robustly and interactively, which is a very important and advantageous advantage in DSL.

[0066] In this embodiment, the file to be parsed and the module file can be written in programming languages ​​such as Python, Java, C and C#, preferably written in TAF language, and parsed into shell language by a compiler.

[0067] TAF file refers to the source code file of TAF, which is used to store TAF code in plain text form. The file contains functions, classes, variable definitions and execution logic, and is the main place for writing programs.

[0068] TAF is a DSL (Domain Specific Language) designed for terminal command line operations (Shell) to facilitate containerized work and process design and management. TAF is case-sensitive and indentation-insensitive. It is separated only by line breaks, and comments start with #. The grammatical structure of the language is a header system, with three layers of identifiers from top to bottom:

[0069] 1) Top-level identifier: located in the first line of each file and unique, used to mark the type (tool or workflow) of the taf file and give it a name, such as +TOOL:xxx / +FLOW:xxx;

[0070] 2) First-level identifier: such as LOAD / ARGS / RUN (RUN:PRE / RUN / RUN:AFTER), etc. LOAD is used to call TAF files, ARGS is used to set parameters, and RUN is used to set the running content. There can be only one first-level identifier in each file. If there are two, the first one takes precedence and all subsequent identifiers with the same name are ignored;

[0071] 3) Secondary identification: <xxx>, enclosed in angle brackets, the specific content and purpose depends on the first-level identifier, and the content and usage of different first-level identifiers are different. For example, the second-level identifier under LOAD is used to fill in the path or alias of the TAF file, which can be expressed as:

[0072] LOAD

[0073]

[0074] . / tools / a.taf;

[0075] The secondary identifier under ARGS is used to define the value of the variable, and the identifier supports cross-line parameters. The variable replacement uses ::xxx:: to call the parameter, and the call supports cross-file calls, such as using LOAD to call a.taf file in a taf file, and using As an alias of a.taf, you can use ::a:xxx:: to access the xxx variable in a.taf and replace the variable before running;

[0076] It also provides built-in environment variables to help developers quickly build taf workflow templates, such as ::*WORKDIR*:: indicates the current working path; ::*LOAD-DIR*:: indicates the path of the taf file being run; ::*CPUS*:: indicates the number of CPU threads on the current operating platform; ::*MAIN*:: is a variable with a built-in if command for quick tool building. If the user does not pass in parameters, then if=::else::. If there are parameters passed in and they start with -xxx, the official build method is used, if=::main::. If it does not start with -xxx, the user passes in a new command, which is passed in directly. If there is a taf file:

[0077] +TOOL:blast

[0078] ARGS

[0079] <mian>

[0080] ::cmd::::opts::::in::

[0081] <else>

[0082] cat::*LOAD-DIR*::.. / blast.info .txt

[0083] RUN

[0084] <container:taf-blast: docker.io / ncbi / blast:latest >

[0085] ::*MAIN*::

[0086] For example, the taf file mentioned above can be used in the following three ways:

[0087] (1) taf-blast: If the user does not pass in any parameters, ::else:: is called, which displays help;

[0088] (2) taf-blast—cmd blastn—in xxx—opts yyy: The user inputs parameters, and the command starts with —cmd, and runs by calling ::main::.

[0089] (3) taf-blast blastn xxx yyy…: The user inputs parameters that do not begin with —xxx. blastnxxx yyy… is directly input to the ::*MAIN*:: position.

[0090] RUN is the code of all the commands that need to be executed in the taf file. RUN can be divided into three identifiers: RUN:PRE / RUN / RUN:AFTER, and the contents of RUN will be executed in this order. There are five main secondary identifiers of RUN:

[0091] 1) <local sh shell>: Corresponding to local shell code, you can write shell code under this mark, and this part of the shell code will be directly copied without modification during conversion;

[0092] 2)<container / apptainer / singularity / docker / podman:container-name:image-name(@...$...)> : Corresponding to the code running in containers such as apptainer, docker or podman, when the label is recognized, the available containers and images are automatically detected locally. If there are multiple containers and images locally, the order of installation can be specified in the configuration file, such as the congif configuration file, or the first detected container or image can be installed. If the corresponding image and container do not exist locally, the compiler will automatically generate code to obtain the image and generate the container to automatically configure the environment. If there is one, the corresponding container is used directly;

[0093] 3)<python / R / …> : Write the code in the corresponding language, pass it to the corresponding compiler, and then output the running results;

[0094] 4)<sh:…> : This flag can use the corresponding command after sh as the shell command header, and then pass the code below sh into the command through heredoc;

[0095] 5) <flow>:The core identifier of the workflow, automatically identifies the taf command in it, and replaces it with a shell command in pre-run, and then builds the flow process by writing a shell.

[0096] In this embodiment, the parsing unit is further used to: based on the pre-run parameters, run the first script data to obtain output parameters, and based on the first file calling sequence, splice the output parameters to obtain the run instruction;

[0097] For example, at any position in the taf file, you can use the pre-run parameter $$command$$ to wrap the script data in the shell and use its output as the value corresponding to the script data. For example, you can use it in RUN- <local>Use $$taf-a–dry-run…$$, $$taf-b–dry-run…$$ to pre-call different taf files, and embed the converted shell code into the current taf file. This can achieve a fast build process like writing shell and ensure that there are no problems such as environment conflicts.

[0098] In a more preferred embodiment, the RUN <flow>Tags are optimized, and will automatically recognize "taf-xxx" and "taffish xxx.taf" commands, and replace them with corresponding shell texts after pre-running, without manually adding $$...$$; and provide <auto-flow>Tags, which are <flow>On the basis of , it will automatically identify and install the taf-xxx that appears, and run "taf install-n xxx" in advance, that is, automatically identify and install dependencies.

[0099] Using the TAF language, which is almost identical to Shell syntax, means that users no longer need to learn complex programming languages ​​or master new technologies. For users, being familiar with Shell commands can quickly build workflows, which can shorten the time of workflow development. At the same time, the original shell script can be converted into a TAF script in just two steps with minimal cost: (1) Add a TAF header to the front of the original script and use <flow>or <auto-flow>Tag; (2) Add taf-xxx in front of all taf tools to adapt to the environment. Especially for bioinformaticians and researchers with non-programming backgrounds, compared with Nextflow's Groovy language or Snakemake's complex rule definitions, the learning curve is greatly reduced.

[0100] In this embodiment, the parsing unit is further used to: convert the development workflow template into an executable file, and obtain the development workflow template based on the executable file. For example, PyInstaller is a popular Python code packaging tool that can package Python scripts into executable files.

[0101] You can directly install the xxx module using taf install xxx and other methods, and then directly call this module using taf-xxx, and you can also directly view the help and usage of this module using taf-xxx-h. This is more in line with the usage of the shell itself and reduces the learning cost.

[0102] In this embodiment, the system can run on different operating systems or hardware platforms, that is, it can run across platforms.

[0103] Embodiment 2

[0104] refer to < / flow> < / flow> < / flow> < / local> < / flow> < / local> < / else> < / mian> Figure 1-Figure 2 Based on the first embodiment, in this embodiment, the parsing unit is further used to obtain a development workflow template, and the method of obtaining the development workflow template is:

[0105] The compiler obtains several second files of the module file, obtains fourth script data of each second file based on predefined module parameters, splices the fourth script data based on the calling order of the module files to obtain development instructions; and obtains the development workflow template based on the development instructions.

[0106] The calling unit is further used to upload the development workflow template to the cloud warehouse and / or the local warehouse.

[0107] If the module file includes four files: m, n, q and p, then based on <flow>Obtain the four files, refer to the method of obtaining the running instructions, and the compiler parses them to obtain the script data. According to the calling order, the script data is assembled to obtain the development instructions, and they are packaged into a module package to obtain the development workflow template.

[0108] Embodiment 3

[0109] On the basis of the above embodiment, in this embodiment, each of the cloud workflow templates and each of the local workflow templates corresponds to a template data, and the template data includes a unique identifier, an upload time, an upload user, a distinguishing parameter, an upload development parameter, and an upload environment parameter;

[0110] Each of the development workflow templates corresponds to a development data, and the development data includes development time, development user, the development instruction and development environment parameters.

[0111] The system further includes a distinguishing unit, which is used to obtain the distinguishing parameter. The distinguishing parameter is obtained in the following manner:

[0112] Obtaining environment differentiation data based on the uploaded environment parameters, the environment differentiation data including operating system, application service, programming language, framework required by programming language and environment dependencies required by programming language, database and application service;

[0113] Obtaining encoding difference data based on the uploaded development parameters, the encoding difference data including the number of files, the order of file calls and SQL statements;

[0114] The distinguishing parameter is obtained based on the environment distinguishing data and the encoding distinguishing data.

[0115] In this embodiment, the application service may include a database, an application server, a message queue, a cache service, a log monitoring tool, and the like.

[0116] The system further includes a prediction unit, which is used to obtain a prediction workflow template. The prediction workflow template is obtained in the following manner:

[0117] Acquire historical operation data of the user, obtain a work category, a template usage frequency and a template usage time based on the historical operation data, and obtain a first workflow template list based on the work category;

[0118] Acquire real-time operation time, obtain a time range based on the real-time operation time, and obtain a second workflow template list based on the time range, the template use time, and the first workflow template list;

[0119] Based on the template usage frequency and the second workflow template list, obtaining a third workflow template list;

[0120] Based on the preset list range and the third workflow template list, the predicted workflow template is obtained.

[0121] For example, if the current time is 10 o'clock, first obtain the user's work category, and based on the work category, obtain the first workflow template list used by the user. Based on the workflow template list, obtain the workflow templates used by the user in the time period of 9-11 am within one month, obtain the second workflow template list, obtain the usage frequency of each template in the second workflow template list, select the top 2 workflow templates with the highest usage frequency, and obtain the predicted workflow template.

[0122] Embodiment 4

[0123] On the basis of the above embodiments, in this embodiment, the system further includes a similarity unit, which is used to obtain a workflow template grouping, obtain a target similarity based on the workflow template grouping, the operating parameters and the operating environment data, obtain an optimal workflow template based on the target similarity, and update the predicted workflow template to the optimal workflow template; compare the operating parameters and the operating environment data with the workflow template grouping to obtain a template with the highest similarity and obtain the optimal workflow template.

[0124] The workflow template grouping is obtained as follows:

[0125] Based on the environment distinction data, a first similarity is obtained; based on the number of files and the file calling order, a second similarity is obtained; based on the SQL statement, a third similarity is obtained; based on the first similarity, the second similarity and the third similarity, a total similarity is obtained, and based on the total similarity, the cloud workflow template and the local workflow template are divided to obtain the workflow template group.

[0126] For example, the same number of operating systems, application services, programming languages, frameworks, and environment dependencies is obtained, and the first similarity is calculated, where the first similarity = the same number / total number; the number of files is obtained to obtain the file similarity, the file calling order is obtained to obtain the sequence similarity, different weights are assigned to them, and the second similarity is obtained by adding them up; the identifiers in the SQL statement and the number and order of the identifiers are obtained, the similarities of the identifiers, number, and order are obtained respectively, different weights are assigned to them, and the third similarity is obtained by adding them up; different weights are assigned, the first similarity, the second similarity, and the third similarity are added together to obtain the total similarity, and the total similarity is divided into different ranges, thereby obtaining the workflow template grouping.

[0127] Alternatively, the data may be divided first according to the subdivided data under the environment distinguishing data and the coding distinguishing data, and then the similarity may be calculated. For example, the data may be divided first according to the operating system, and then the similarity may be calculated in sequence.

[0128] Embodiment 5

[0129] On the basis of the above embodiment, in this embodiment, the similar unit is also used for:

[0130] Based on the environment difference data and the development environment parameter, obtaining a first development similarity;

[0131] obtaining a second development similarity based on the development instruction and the encoding difference data;

[0132] Obtaining a similar workflow template based on the first development similarity, the second development similarity, and the workflow template grouping;

[0133] Referring to the method for calculating similarity in Example 4, the first development similarity and the second development similarity are obtained, and they are also compared with the workflow template grouping, and the template with the highest similarity is obtained as the similar workflow template;

[0134] Acquire similar environment parameters and similar development instructions of the similar workflow template; acquire user modification data;

[0135] Based on the user modified data, the similar environment parameters and the similar development instructions, a new workflow template is obtained, and the parameters modified by the user are replaced to obtain a new workflow template.

[0136] Embodiment 6

[0137] On the basis of the above embodiment, in this embodiment, the system further includes the same unit, and the same unit is used for:

[0138] Determine whether the total similarity is greater than a first preset similarity, and if so, obtain a plurality of first identical workflow templates based on the total similarity, obtain the same unique identifier of all the first identical workflow templates, and transmit the same unique identifier to the backend for processing;

[0139] And based on the first development similarity and the second development similarity, a third development similarity is obtained, and it is determined whether the third development similarity is greater than a second preset similarity. If so, a plurality of second identical workflow templates are obtained based on the third development similarity, and all the second identical workflow templates are returned to the user end for processing.

[0140] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0141] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.< / flow> < / xxx> < / local> < / flow> < / run> < / workdir>

Claims

1. A cross-platform workflow development and calling system based on modular management, characterized in that: The system comprises: Acquisition unit: used to obtain files to be parsed, module files and command parameters; Parsing unit: used to obtain running instructions; Cloud warehouse unit: used to store cloud workflow templates based on cloud warehouse; Local warehouse unit: used to store local workflow templates based on the local warehouse; A calling unit: used for inputting the running instruction into a transmission tool, wherein the transmission tool obtains a workflow identifier based on the running instruction, obtains the cloud-based workflow template and / or the local workflow template based on the workflow identifier, and obtains a target workflow template; The operation instruction is obtained as follows: Based on the predefined parsing tag, the compiler obtains the parsing parameters of the file to be parsed, replaces the predefined parameters based on the parsing parameters and the command parameters, and obtains the operating parameters and the operating environment data; The environment type is obtained based on the environment tag of the operating environment data, and the operating instruction is obtained based on the operating parameter and the environment type.

2. According to claim 1, a cross-platform workflow development and calling system based on modular management is characterized in that: The specific steps of obtaining the operating instruction based on the operating parameters and the environment type include: If the environment type is a process-based type, obtaining a plurality of first files based on the running parameters, obtaining first script data in each of the first files, and splicing the first script data based on the calling sequence of the first files to obtain the running instructions; If the environment type is an isolated type, generating second script data of an isolated environment, compiling the operating parameters into third script data of the isolated environment, and obtaining the operating instructions based on the second script data and the third script data; If the environment type is a normalized type, the operating instruction is obtained based on the operating parameters.

3. According to the cross-platform workflow development and calling system based on modular management according to claim 1, it is characterized in that: The parsing unit is further used to obtain a development workflow template, and the method of obtaining the development workflow template is: The compiler obtains a plurality of second files of the module file, obtains fourth script data of each second file based on predefined module parameters, and splices the fourth script data based on the calling order of the module files to obtain development instructions; The development workflow template is obtained based on the development instruction.

4. According to claim 3, a cross-platform workflow development and calling system based on modular management is characterized in that: The calling unit is further used to upload the development workflow template to the cloud warehouse and / or the local warehouse.

5. A cross-platform workflow development and calling system based on modular management according to claim 4, characterized in that: Each of the cloud workflow templates and each of the local workflow templates corresponds to a template data, and the template data includes a unique identifier, an upload time, an upload user, a distinguishing parameter, an upload development parameter, and an upload environment parameter; Each of the development workflow templates corresponds to a development data, and the development data includes development time, development user, the development instruction and development environment parameters.

6. A cross-platform workflow development and calling system based on modular management according to claim 5, characterized in that: The system further includes a distinguishing unit, which is used to obtain the distinguishing parameter. The distinguishing parameter is obtained in the following manner: Obtaining environment differentiation data based on the uploaded environment parameters, the environment differentiation data including operating system, application services, programming language, framework and environment dependencies; Obtaining encoding difference data based on the uploaded development parameters, the encoding difference data including the number of files, the order of file calls and SQL statements; The distinguishing parameter is obtained based on the environment distinguishing data and the encoding distinguishing data.

7. A cross-platform workflow development and calling system based on modular management according to claim 6, characterized in that: The system further includes a prediction unit, which is used to obtain a prediction workflow template. The prediction workflow template is obtained in the following manner: Acquire historical operation data of the user, obtain a work category, a template usage frequency and a template usage time based on the historical operation data, and obtain a first workflow template list based on the work category; Acquire real-time operation time, obtain a time range based on the real-time operation time, and obtain a second workflow template list based on the time range, the template use time, and the first workflow template list; Based on the template usage frequency and the second workflow template list, obtaining a third workflow template list; Based on the preset list range and the third workflow template list, the predicted workflow template is obtained.

8. A cross-platform workflow development and calling system based on modular management according to claim 7, characterized in that: The system further includes a similarity unit, which is used to obtain a workflow template group, obtain a target similarity based on the workflow template group, the operating parameters and the operating environment data, obtain an optimal workflow template based on the target similarity, and update the predicted workflow template to the optimal workflow template; The workflow template grouping is obtained as follows: Based on the environment distinguishing data, a first similarity is obtained; based on the number of files and the order in which the files are called, a second similarity is obtained; A third similarity is obtained based on the SQL statement; a total similarity is obtained based on the first similarity, the second similarity and the third similarity; the cloud workflow template and the local workflow template are divided based on the total similarity to obtain the workflow template group.

9. A cross-platform workflow development and calling system based on modular management according to claim 8, characterized in that: The similar units are also used for: Based on the environment difference data and the development environment parameter, obtaining a first development similarity; obtaining a second development similarity based on the development instruction and the encoding difference data; Obtaining a similar workflow template based on the first development similarity, the second development similarity, and the workflow template grouping; Acquire similar environment parameters and similar development instructions of the similar workflow template; Get user modification data; A new workflow template is obtained based on the user modification data, the similar environment parameters and the similar development instructions.

10. A cross-platform workflow development and calling system based on modular management according to claim 9, characterized in that: The system further comprises the same unit, wherein the same unit is used for: Determine whether the total similarity is greater than a first preset similarity, and if so, obtain a plurality of first identical workflow templates based on the total similarity, obtain the same unique identifier of all the first identical workflow templates, and transmit the same unique identifier to the backend for processing; And based on the first development similarity and the second development similarity, a third development similarity is obtained, and it is determined whether the third development similarity is greater than a second preset similarity. If so, a plurality of second identical workflow templates are obtained based on the third development similarity, and all the second identical workflow templates are returned to the user end for processing.

Images