Implementation method and equipment of scientific workflow engine and medium
Through the implementation methods of scientific workflow engine, including data management, scientific computing and workflow management modules, the problem of lack of unified standards and compatibility in the existing technology is solved, efficient data and computing management is realized, and the system capability is improved Scalability and maintainability.
Patent Information
- Application Number
- CN202411051368.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-22
AI Technical Summary
Most existing scientific workflow engines are designed for specific business areas, lack of unified standards, and the use of different modeling languages leads to difficulties in porting and scaling, and lacks cross-domain and cross-platform compatibility.
Provides a method for implementing a scientific workflow engine, including data management module, scientific computing module and workflow management module, manages intensive data through abstract interfaces, distributed storage and indexing mechanisms, calculate component libraries and preset methods to call computing resources , scientific workflow virtual machines and activity models, hierarchical architecture and computing component registration mechanisms, realize efficient management of data and computing and flexible management of workflows.
Improve data processing and computing efficiency, enhance data scalability and access speed, realize efficient execution of scientific computing and accurate management of workflows, simplify workflow modeling and execution, improve system maintainability and Extensibility.
Smart Images

Figure CN119003117A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method, device and medium for implementing a scientific workflow engine. Background Art
[0002] At present, in the current scientific research field, with the continuous expansion of data scale and the significant increase in complexity, scientists are facing unprecedented challenges. They need to process large-scale data sets, perform complex analyses, and ensure the efficiency and repeatability of the research process. In order to effectively meet these challenges, scientific workflows have emerged as an important tool. Scientific workflows can comprehensively coordinate and manage all aspects of scientific research, from the collection and preprocessing of raw data to in-depth data analysis and the final presentation of results, so that scientists can carry out research more efficiently and systematically.
[0003] As the core component of scientific workflow, the scientific workflow engine plays a vital role. It is responsible for parsing workflow definition files and accurately controlling the flow of business processes according to the business interaction logic. Through advanced scheduling algorithms, the scientific workflow engine can manage various states of the process, such as start, stop, suspend, and resume, and handle complex interactions in each link of the process, such as link merging, forking, selection, and conditional selection.
[0004] In the field of business process modeling, Business Process Modeling Notation (BPMN) is a widely accepted modeling language standard that provides strong support for building clear and standard business process diagrams. BPMN is created and maintained by the Object Management Group (OMG). It has gone through multiple versions of development, among which the BPMN 2.0 standard was launched in 2011 and became the industry standard. The main advantage of BPMN is that it provides a set of markup languages that are easy for all business users to understand. Whether it is business analysts, software developers or business managers, they can accurately understand and communicate business processes through BPMN. It builds a standardized bridge between the design and implementation of business processes, greatly promoting the standardization and normalization of business processes.
[0005] However, most of the current scientific workflow engines are designed and developed for specific business fields, lacking unified standards and specifications. These engines usually use UML activity diagrams or Petri nets for model definition, which can meet the needs of specific fields but bring difficulties in porting and expansion. The lack of compatibility between different scientific workflow engines has caused scientists to face many inconveniences when conducting cross-domain or cross-platform research. Summary of the invention
[0006] The embodiments of the present application provide a method, device and medium for implementing a scientific workflow engine, so as to solve the technical problems that most existing scientific workflow engines are designed for specific business fields, lack unified standards, use different modeling languages, and thus are difficult to transplant and expand.
[0007] On the one hand, an embodiment of the present application provides a method for implementing a scientific workflow engine, which is applied to an implementation system of a scientific workflow engine, wherein the system includes a data management module, a scientific computing module, and a workflow management module, including:
[0008] Based on the data management module, manage and control the workflow corresponding to the intensive data;
[0009] Based on the scientific computing module, perform scientific computing on the intensive data;
[0010] Based on the workflow management module, the intensive data is stored and managed.
[0011] In one implementation of the present application, the data management module includes an abstract interface;
[0012] The method further comprises:
[0013] Based on the abstract interface, performing a read operation or a write operation on the intensive data;
[0014] Dividing the dense data to obtain corresponding divided data;
[0015] Distributed storage of the partitioned data is performed through a distributed file system, and an index corresponding to each partitioned data is determined;
[0016] Based on the index corresponding to the partitioned data, data access is performed on the partitioned data in the distributed file system.
[0017] In one implementation of the present application, the scientific computing module includes a computing component library;
[0018] The method further comprises:
[0019] Determine the data to be calculated and the computing resources to be called corresponding to the data to be calculated;
[0020] Calling the computing resource in the computing component library in a preset manner; wherein the preset manner includes HTTP manner and RPC manner;
[0021] Based on the computing resources, a computing process corresponding to the data to be calculated is completed.
[0022] In one implementation of the present application, the scientific computing module further includes a computing component registration mechanism;
[0023] The method further comprises:
[0024] Determine the developed computing components, and fill in the component information corresponding to the computing components based on the computing component registration mechanism; wherein the component information includes name, description, input parameters, output parameters, calling method, and whether parallel computing is supported;
[0025] Based on the component information, the scientific workflow engine is allowed to call the computing component.
[0026] In one implementation of the present application, the workflow management module includes a scientific workflow virtual machine;
[0027] The method further comprises:
[0028] An abstract API interface corresponding to the scientific workflow virtual machine is determined, and at least one workflow operation condition is determined through the abstract API interface.
[0029] In one implementation of the present application, the workflow management module further includes a scientific workflow activity model;
[0030] The method further comprises:
[0031] Determine a model class library corresponding to the scientific workflow activity model, and based on the model class library, determine activity attributes corresponding to each activity class in the scientific workflow;
[0032] The activity attributes include start, end, calculation, conversion, message and sequence flow.
[0033] In one implementation of the present application, the workflow management module further includes a scientific workflow engine;
[0034] The method further comprises:
[0035] Based on the scientific workflow engine, the specific behavior of each activity class in the scientific workflow activity model is determined, and the corresponding activity instance class is determined.
[0036] In one implementation of the present application, the method further includes:
[0037] The scientific workflow engine is layered to determine multiple levels corresponding to the scientific workflow engine; wherein the multiple levels include: a core interface layer, a basic component layer, a core implementation layer, a command layer, a command interception layer, a business interface layer and a process engine;
[0038] Determine a packaging method corresponding to the multiple levels of the layered architecture; wherein the packaging method is bottom-up packaging.
[0039] On the other hand, the embodiment of the present application further provides a scientific workflow engine implementation device, which is applied to a scientific workflow engine implementation system, the system includes a data management module, a scientific computing module and a workflow management module, and the device includes:
[0040] at least one processor;
[0041] and, a memory communicatively coupled to the at least one processor;
[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the implementation method of a scientific workflow engine as described above.
[0043] On the other hand, an embodiment of the present application also provides a non-volatile computer storage medium storing computer executable instructions, which is applied to an implementation system of a scientific workflow engine, wherein the system includes a data management module, a scientific computing module, and a workflow management module. When the computer executable instructions are executed, a method for implementing a scientific workflow engine as described above is implemented.
[0044] The embodiments of the present application provide a method, device and medium for implementing a scientific workflow engine, which at least have the following beneficial effects:
[0045] Through the data management module and the scientific computing module, effective management and scientific computing of intensive data are achieved, and data processing and computing efficiency are improved; through the workflow management module, storage and management of intensive data are achieved, which facilitates data access and use; through abstract interfaces, intensive data are operated, the flexibility and scalability of data processing are improved, and the distributed storage and indexing mechanism improves data access speed and processing efficiency; through the computing component library and preset methods to call computing resources, efficient execution of scientific computing is achieved, and by supporting multiple preset methods, such as HTTP, RPC, the flexibility and compatibility of calling computing resources are improved; through the computing component registration mechanism, newly developed computing components can be easily registered in the scientific workflow engine, which improves the scalability of the system, and through component information, the scientific workflow engine can accurately call the required computing components, which improves the accuracy and efficiency of the computing process; through the scientific workflow virtual machine It provides an abstract API interface, making workflow operations more flexible and convenient, and determines workflow operation status through the abstract API interface, thereby improving the efficiency and accuracy of workflow management; through the scientific workflow activity model and its model class library, it provides a unified description and management method for each activity class in the scientific workflow. Through activity attributes, it can accurately describe the behavior and characteristics of each activity in the scientific workflow, thereby improving the accuracy and efficiency of workflow modeling; through the scientific workflow engine, it can determine the specific behavior of each activity class in the scientific workflow activity model, and realize the automatic execution and management of the workflow. Through the activity instance class, it can easily create and manage each activity instance in the scientific workflow, thereby improving the flexibility and efficiency of workflow execution; through the layering of the scientific workflow engine, the structure of the engine is clearer and easier to manage, and through the bottom-up packaging method, the dependency relationship between each level is clearer, thereby improving the maintainability and scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0047] Figure 1 A flowchart of a method for implementing a scientific workflow engine provided in an embodiment of the present application;
[0048] Figure 2 A schematic diagram of the internal structure of a device for implementing a scientific workflow engine provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0050] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0051] Figure 1 A flowchart of a method for implementing a scientific workflow engine provided in an embodiment of the present application.
[0052] The analysis method involved in the embodiments of the present application can be implemented by a terminal device or a server, and the present application does not impose any special restrictions on this. For the convenience of understanding and description, the following embodiments are described in detail by taking a server as an example.
[0053] It should be noted that the server may be a single device or a system consisting of multiple devices, that is, a distributed server, and this application does not make any specific limitation on this.
[0054] like Figure 1 As shown, a method for implementing a scientific workflow engine provided in an embodiment of the present application is applied to an implementation system of a scientific workflow engine, the system including a data management module, a scientific computing module and a workflow management module, including:
[0055] 101. Based on the data management module, manage and control the workflow corresponding to intensive data;
[0056] Specifically, in one embodiment of the present application, the data management module includes an abstract interface. Based on the abstract interface, the server performs a read operation or a write operation on the dense data; divides the dense data to obtain corresponding divided data; distributes the divided data through a distributed file system and determines the index corresponding to each divided data; and accesses the divided data in the distributed file system based on the index corresponding to the divided data.
[0057] In one embodiment, first, an abstract interface of a data management module is defined, which provides a method for reading and writing intensive data, thereby realizing customized and personalized data management. Users can implement customized data management logic based on these abstract interfaces according to actual needs. By implementing the abstract interface of the data management module, intensive data can be read and written. These operations may involve obtaining data from an external data source, or writing calculation results into persistent storage.
[0058] In order to improve data processing efficiency, intensive data is divided to obtain multiple partitioned data. These partitioned data can be divided according to the characteristics, size or processing requirements of the data for subsequent distributed storage and processing. Using the distributed file system, the partitioned data is stored on multiple nodes in the cluster to achieve distributed storage of data. At the same time, a corresponding index is created for each partitioned data to quickly locate and access the data.
[0059] When you need to access a certain partition of data, you can quickly locate it in the distributed file system through its corresponding index and read the required data. This index-based data access method improves data access speed and efficiency.
[0060] In one embodiment, the data storage implemented by default uses a distributed file system to store large-scale intensive data. The data is divided into multiple blocks and distributedly stored in a cluster to improve data processing efficiency and reliability.
[0061] Data access adopts an index-based data access method. By establishing metadata indexes, data can be quickly located and accessed, reducing the time overhead of data reading and writing. The data management module also supports data backup, recovery and data consistency management. It ensures data reliability and fault tolerance through a data backup mechanism, and ensures data consistency between multiple computing tasks through a consistency management mechanism.
[0062] 102. Based on the scientific computing module, perform scientific computing on intensive data.
[0063] Specifically, in one embodiment of the present application, the scientific computing module includes a computing component library. The server determines the data to be calculated and the computing resources to be called corresponding to the data to be calculated; the computing resources are called in the computing component library in a preset manner. It should be noted that the preset manner in the embodiment of the present application includes HTTP and RPC, and the computing component library provides an interface for calling a specified computing component.
[0064] Then, the server completes the calculation process corresponding to the data to be calculated based on the computing resources.
[0065] In one embodiment, in the scientific workflow engine, the scientific computing module is a core part, which is responsible for performing complex calculations on intensive data. In order to achieve efficient and flexible calculations, the scientific computing module includes a computing component library, which stores various computing resources and algorithms.
[0066] First, the system needs to determine which data needs to be scientifically computed and identify the computing resources corresponding to these data. These computing resources may be predefined algorithms, models, or functions, which are stored in the computing component library. Once the data to be calculated and the corresponding computing resources are determined, the system can call these resources in the computing component library in a preset way. The preset calling method may include HTTP request, remote procedure call (RPC) or other communication protocols.
[0067] After calling computing resources, the system can perform the calculation process corresponding to the data to be calculated based on these resources. This may involve steps such as data preprocessing, algorithm execution, and result generation. After the calculation is completed, the system will process the generated results, which may include storing the results in a database, sending the results to other modules for further processing, or returning the results to the user.
[0068] Suppose you are using a scientific workflow engine to process astronomical data and need to calculate the age of a star. In this scenario, the data to be calculated is the observational data of the star, and the computing resource to be called is a star age calculation algorithm stored in the computing component library.
[0069] The system first determines the data to be calculated and the corresponding computing resources, and then calls the star age calculation algorithm in the computing component library through HTTP requests. After the algorithm is executed, the system will process the generated results and store the star age information in the database or return it to the user.
[0070] In one embodiment of the present application, the scientific computing module also includes a computing component registration mechanism. The server determines the computing components that have been developed, and fills in the component information corresponding to the computing components based on the computing component registration mechanism. It should be noted that the component information in the embodiment of the present application includes name, description, input parameters, output parameters, calling method, and whether parallel computing is supported.
[0071] The server then allows the scientific workflow engine to call the computational components based on the component information.
[0072] In one embodiment, in order to improve the flexibility and scalability of the computing module, a computing component registration mechanism is introduced. Through this mechanism, developers can register newly developed computing components into the scientific computing module and fill in relevant component information so that the scientific workflow engine can call these components to perform computing tasks.
[0073] First, developers need to develop a new computing component, which may be an algorithm, model or function used to perform specific scientific computing tasks. After the development is completed, developers need to fill in the component information corresponding to the computing component based on the computing component registration mechanism. This information includes the component name, description, input parameters (input parameters), output parameters (output parameters), calling method, and whether parallel computing is supported.
[0074] After filling in the component information, the developer registers the computing component to the scientific computing module. This usually involves submitting the component code and related information to the management system of the scientific workflow engine. Once the computing component is successfully registered, the scientific workflow engine can call the component to perform computing tasks based on the component information. The engine will pass the corresponding data to the component and obtain the output results of the component based on the input parameters and calling method of the component.
[0075] If the computing component supports parallel computing, the scientific workflow engine can also use this feature to distribute computing tasks to multiple computing nodes for parallel execution to improve computing efficiency.
[0076] Suppose you are developing a scientific workflow engine for image processing. The developer has developed a new image filtering algorithm and wants to register it in the scientific computing module. The developer first develops the image filtering algorithm and fills in the component information of the algorithm based on the computing component registration mechanism, including the algorithm's name, description, input image and parameters, output image, calling method, and whether parallel computing is supported. Then, the developer registers the image filtering algorithm in the scientific computing module. When the scientific workflow engine performs image processing tasks, it can call the algorithm to filter the input image and obtain the processed output image.
[0077] 103. Store and manage intensive data based on workflow management module.
[0078] Specifically, in one embodiment of the present application, the workflow management module includes a scientific workflow virtual machine. The server determines an abstract API interface corresponding to the scientific workflow virtual machine, and determines at least one workflow operation condition through the abstract API interface.
[0079] In one embodiment, the scientific workflow virtual machine SwfVM is a set of abstract API interfaces without specific implementation, and various possibilities in workflow are described through the API itself. Without specific implementation, SwfVM itself can better adapt to various different workflow domain languages, and the workflow engine itself is also an implementation on SwfVM.
[0080] In one embodiment, in the scientific workflow engine, the workflow management module is a core component, which is responsible for managing and executing scientific workflows. In order to improve the flexibility and scalability of the workflow, the workflow management module introduces a scientific workflow virtual machine, which can simulate and execute various scientific workflows.
[0081] First, the system needs to identify the scientific workflow virtual machine, which is a key component in the workflow management module and is used to execute and manage scientific workflows. The scientific workflow virtual machine provides a series of abstract API interfaces that define standard ways to interact with the virtual machine. The system needs to identify these abstract API interfaces so that it can communicate with the virtual machine through them later.
[0082] Through the abstract API interface, the system can determine at least one workflow operation. These operation conditions may include starting, stopping, pausing, resuming, restarting, etc. of the workflow. The system needs to send corresponding operation instructions to the scientific workflow virtual machine through the abstract API interface according to actual needs. Once the workflow operation condition is determined, the system can send operation instructions to the scientific workflow virtual machine through the abstract API interface. After receiving the instruction, the virtual machine will perform the corresponding operation and return the operation result to the system. After receiving the operation result returned by the scientific workflow virtual machine, the system will process the result. This may include storing the result in a database, sending the result to other modules for further processing, or returning the result to the user.
[0083] Suppose that a scientific workflow engine is being used to execute an astronomy data analysis workflow. In this scenario, the scientific workflow virtual machine is responsible for executing and managing the entire workflow. First, the system determines the scientific workflow virtual machine and identifies the abstract API interface provided by the virtual machine. Then, the system determines the workflow operation that needs to be performed through the abstract API interface, such as starting the workflow.
[0084] Next, the system sends a startup instruction to the scientific workflow virtual machine through the abstract API interface. After receiving the instruction, the virtual machine will start the astronomical data analysis workflow and perform a series of calculation and analysis tasks. Finally, the scientific workflow virtual machine returns the execution results to the system, which processes the results and displays the analysis results to the user or stores them in the database.
[0085] In one embodiment of the present application, the workflow management module also includes a scientific workflow activity model. The server determines a model class library corresponding to the scientific workflow activity model, and based on the model class library, determines the activity attributes corresponding to each activity class in the scientific workflow. It should be noted that the activity attributes in the embodiment of the present application include start, end, calculation, conversion, message and sequence flow.
[0086] In one embodiment, the scientific workflow activity model SwfModel is a set of model class libraries implemented based on the BPMN2.0 specification, which is used to describe the properties of various activities in the scientific workflow, including start SwfStart, end SwfEnd, calculation SwfCompution, conversion SwfTranslation, message SwfMessage, sequence flow SwfSequenceFlow, etc.
[0087] In one embodiment, in a scientific workflow engine, the workflow management module is not only responsible for managing and executing scientific workflows, but also needs to define and describe the various activities in these workflows. In order to improve the descriptive power and flexibility of the workflow, the workflow management module introduces a scientific workflow activity model, which can define and describe various activities and their attributes in the scientific workflow.
[0088] First, the system needs to determine the scientific workflow activity model, which is a key component in the workflow management module and is used to define and describe various activities in the scientific workflow. The scientific workflow activity model corresponds to a model class library, which contains templates and properties that define each activity class in the scientific workflow. The system needs to identify this model class library so that it can subsequently determine the activity properties of the activity class based on it.
[0089] Through the model class library, the system can determine the activity classes and their corresponding activity attributes in the scientific workflow. These activity attributes include start, end, calculation, conversion, message and sequence flow, which describe the start and end conditions of the activity, the calculation process, data conversion, message passing and the sequential relationship between activities. Once the activity class and its activity attributes are determined, the system can configure these attributes. This includes setting the start and end conditions of the activity, defining the calculation process and data conversion rules, configuring the message passing mechanism and determining the sequence flow between activities. After configuring the activity attributes, the system can execute and monitor the scientific workflow. During the execution process, the system will control the start and end of the activity, execute the calculation process, perform data conversion, pass messages and control the sequence flow between activities according to the activity attributes.
[0090] Assume that a bioinformatics data analysis workflow is being executed using a scientific workflow engine. In this scenario, the scientific workflow activity model is responsible for defining and describing the various activities in the workflow. First, the system determines the scientific workflow activity model and identifies the model class library corresponding to the model. Then, the system determines the activity classes and their corresponding activity attributes in the bioinformatics data analysis workflow through the model class library.
[0091] For example, for a data preprocessing activity class, the system determines that its start condition is to receive the original data file, the end condition is to generate the preprocessed data file, the calculation process is to perform a series of data cleaning and conversion operations, and the message passing mechanism is to send the preprocessed data file to the next activity class. After configuring the activity attributes, the system executes and monitors the bioinformatics data analysis workflow. During the execution process, the system will control the start and end of the data preprocessing activity, execute the calculation process, perform data conversion, and pass messages to the next activity according to the activity attributes.
[0092] In one embodiment of the present application, the workflow management module further includes a scientific workflow engine. Based on the scientific workflow engine, the server determines the specific behavior of each activity class in the scientific workflow activity model and determines the corresponding activity instance class.
[0093] In one embodiment, the scientific workflow engine SwfEngine, which is a specific implementation of SwfVM, also defines the specific behaviors (Behavior) and activity instance classes (Instance) of each activity class of SwfModel.
[0094] In one embodiment, in the scientific workflow engine, the workflow management module is responsible for managing and executing the scientific workflow. In order to improve the execution efficiency and flexibility of the workflow, the workflow management module introduces a scientific workflow engine that can interpret and execute the activity classes and their behaviors defined in the scientific workflow activity model.
[0095] First, the system needs to determine the scientific workflow engine, which is a core component in the workflow management module, used to interpret and execute the activity classes and their behaviors defined in the scientific workflow activity model. The system loads the scientific workflow activity model, which describes the various activity classes and their properties in the scientific workflow. These activity classes define the different steps and computational tasks in the workflow. Through the scientific workflow engine, the system can determine the specific behavior of each activity class in the scientific workflow activity model. This includes the execution logic of the activity, the processing of input and output data, and the interaction with other activities.
[0096] Based on the specific behavior of the activity class, the scientific workflow engine generates the corresponding activity instance class. The activity instance class is the specific implementation of the activity class, which contains all the methods and properties required to execute the activity. Once the activity instance class is generated, the scientific workflow engine can execute the scientific workflow. During the execution process, the engine creates objects of the activity instance class and executes the methods of these objects in the order of the workflow to complete the scientific computing task.
[0097] Assume that a chemical simulation workflow is being executed using a scientific workflow engine. In this scenario, the scientific workflow engine is responsible for interpreting and executing the activity classes and their behaviors defined in the chemical simulation workflow. First, the system determines the scientific workflow engine and loads the scientific workflow activity model of the chemical simulation workflow. Then, through the scientific workflow engine, the system determines the specific behaviors of each activity class in the activity model, such as molecular construction, energy calculation, structure optimization, etc.
[0098] Next, the scientific workflow engine generates corresponding activity instance classes based on these specific behaviors, such as molecular construction classes, energy calculation classes, and structure optimization classes. These activity instance classes contain all the methods and properties required to perform chemical simulations. Finally, the scientific workflow engine executes the chemical simulation workflow. During the execution process, the engine creates objects of the activity instance classes and executes the methods of these objects in the order of the workflow to complete the chemical simulation task.
[0099] In one embodiment of the present application, the server layers the scientific workflow engine to determine multiple levels corresponding to the scientific workflow engine. It should be noted that the multiple levels in the embodiment of the present application include: core interface layer, basic component layer, core implementation layer, command layer, command interception layer, business interface layer and process engine.
[0100] Then, the server determines the packaging methods corresponding to the multiple levels of the layered architecture. It should be noted that the packaging method in the embodiment of the present application is bottom-up packaging.
[0101] In one embodiment, the core interface layer, SwfCore, is defined by the SwfVM interface.
[0102] The basic component layer SwfComp provides session context, persistence, event monitoring, cache management, etc.
[0103] The core implementation layer SwfImpl mainly includes SwfModel and related classes. Based on the ideas and interfaces of SwfVM, it defines the key model classes and FlowElementBehavior specific implementations in accordance with the BPMN2.0 specification.
[0104] The command layer SwfCmd uses the command mode to decouple functions and provide an abstract interface. Most of the client requirements of the process engine are implemented externally in the form of specific command implementation classes, such as pass, reject, suspend, resume, enable, disable, etc.
[0105] The command interception layer SwfCmdIncptr adopts the chain of responsibility model and creates conditions for the execution of commands through the interceptor layer of the chain of responsibility model, such as starting transactions, creating command context CommandContext, and recording logs.
[0106] The business interface layer SwfBiz is oriented towards specific businesses and provides various interfaces.
[0107] The process engine SwfEngine is the main entrance to all interfaces. The client can call all APIs provided by the process engine through this interface.
[0108] In one embodiment, in a scientific workflow engine, in order to improve the modularity and scalability of the engine, the engine is usually designed in layers. Each layer is responsible for different functions and responsibilities, and communicates and collaborates through interfaces between layers. At the same time, in order to reduce the coupling between layers, a bottom-up packaging method is usually adopted, that is, each layer provides services and interfaces for the upper layer.
[0109] First, the scientific workflow engine is designed in layers to determine the multiple layers corresponding to the engine. These layers include: core interface layer, basic component layer, core implementation layer, command layer, command interception layer, business interface layer and process engine.
[0110] The following are the responsibilities and functions corresponding to each level identified:
[0111] Core interface layer: defines the core interface and abstract classes of the engine, and provides a unified service access entry for the upper layer.
[0112] Basic component layer: provides the basic components and services required by the engine, such as logging, configuration, database access, etc.
[0113] Core implementation layer: implements the core functions and algorithms of the engine, such as workflow parsing, execution, and monitoring.
[0114] Command layer: defines and executes commands in the engine, such as starting a workflow, stopping a workflow, etc.
[0115] Command interception layer: intercepts and processes commands, such as permission verification, logging, etc.
[0116] Business interface layer: provides business-related interfaces and services, such as scientific computing, data processing, etc.
[0117] Process engine: responsible for the execution and management of the entire workflow, including workflow parsing, task allocation and execution, etc.
[0118] In addition, a bottom-up packaging method is adopted, that is, each layer provides services and interfaces for the layer above it. Specifically, the basic component layer provides services and interfaces for the core implementation layer, the core implementation layer provides services and interfaces for the command layer, the command layer provides services and interfaces for the command interception layer, the command interception layer provides services and interfaces for the business interface layer, and the business interface layer finally provides services and interfaces for the process engine. According to the layered architecture and packaging method, the scientific workflow engine is implemented and deployed. Each layer is implemented according to its responsibilities and functions, and communicates and collaborates with the upper layer through interfaces.
[0119] Suppose that a scientific workflow engine for bioinformatics analysis is being developed. In this scenario, the engine is designed and implemented according to a layered architecture. First, multiple layers of the engine are identified, including the core interface layer, basic component layer, core implementation layer, command layer, command interception layer, business interface layer, and process engine. Then, the responsibilities and functions of each layer are defined, and a bottom-up packaging approach is adopted.
[0120] During the implementation process, we first implemented the basic component layer, which provided basic services such as logging and configuration. Then, we implemented the core implementation layer, which implemented core functions such as workflow parsing and execution. Next, we implemented the command layer and command interception layer, defined the commands for executing workflows, and intercepted and processed the commands. Finally, we implemented the business interface layer and process engine, which provided interfaces and services related to bioinformatics analysis and implemented the execution and management of the entire workflow.
[0121] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, the embodiment of this application also provides an implementation device of a scientific workflow engine, whose structure is as follows: Figure 2 shown.
[0122] Figure 2 The internal structure diagram of a device for implementing a scientific workflow engine provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, it is applied to a scientific workflow engine implementation system, the system includes a data management module, a scientific computing module and a workflow management module, and the device includes:
[0123] at least one processor;
[0124] and, a memory communicatively coupled to the at least one processor;
[0125] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to:
[0126] Based on the data management module, manage and control the workflow corresponding to intensive data;
[0127] Based on the scientific computing module, perform scientific computing on intensive data;
[0128] Store and manage intensive data based on workflow management module.
[0129] The embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, which is applied to a system for implementing a scientific workflow engine. The system includes a data management module, a scientific computing module, and a workflow management module. When the computer executable instructions are executed, they can:
[0130] Based on the data management module, manage and control the workflow corresponding to intensive data;
[0131] Based on the scientific computing module, perform scientific computing on intensive data;
[0132] Store and manage intensive data based on workflow management module.
[0133] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0134] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0135] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.
[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0138] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0140] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0141] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0142] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0143] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0144] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A method for implementing a scientific workflow engine, characterized in that: A system for implementing a scientific workflow engine is applied, the system comprising a data management module, a scientific computing module and a workflow management module, and the method comprises: Based on the data management module, manage and control the workflow corresponding to the intensive data; Based on the scientific computing module, perform scientific computing on the intensive data; Based on the workflow management module, the intensive data is stored and managed.
2. The method for implementing a scientific workflow engine according to claim 1, characterized in that: The data management module includes an abstract interface; The method further comprises: Based on the abstract interface, performing a read operation or a write operation on the intensive data; Dividing the dense data to obtain corresponding divided data; Distributed storage of the partitioned data is performed through a distributed file system, and an index corresponding to each partitioned data is determined; Based on the index corresponding to the partitioned data, data access is performed on the partitioned data in the distributed file system.
3. The method for implementing a scientific workflow engine according to claim 1, characterized in that: The scientific computing module includes a computing component library; The method further comprises: Determine the data to be calculated and the computing resources to be called corresponding to the data to be calculated; Calling the computing resource in the computing component library in a preset manner; wherein the preset manner includes HTTP manner and RPC manner; Based on the computing resources, a computing process corresponding to the data to be calculated is completed.
4. The method for implementing a scientific workflow engine according to claim 1, characterized in that: The scientific computing module also includes a computing component registration mechanism; The method further comprises: Determine the developed computing components, and fill in the component information corresponding to the computing components based on the computing component registration mechanism; wherein the component information includes name, description, input parameters, output parameters, calling method, and whether parallel computing is supported; Based on the component information, the scientific workflow engine is allowed to call the computing component.
5. The method for implementing a scientific workflow engine according to claim 1, characterized in that: The workflow management module includes a scientific workflow virtual machine; The method further comprises: An abstract API interface corresponding to the scientific workflow virtual machine is determined, and at least one workflow operation condition is determined through the abstract API interface.
6. The method for implementing a scientific workflow engine according to claim 1, characterized in that: The workflow management module also includes a scientific workflow activity model; The method further comprises: Determine a model class library corresponding to the scientific workflow activity model, and based on the model class library, determine activity attributes corresponding to each activity class in the scientific workflow; The activity attributes include start, end, calculation, conversion, message and sequence flow.
7. The method for implementing a scientific workflow engine according to claim 1, characterized in that: The workflow management module also includes a scientific workflow engine; The method further comprises: Based on the scientific workflow engine, the specific behavior of each activity class in the scientific workflow activity model is determined, and the corresponding activity instance class is determined.
8. The method for implementing a scientific workflow engine according to claim 7, characterized in that: The method further comprises: The scientific workflow engine is layered to determine multiple levels corresponding to the scientific workflow engine; wherein the multiple levels include: a core interface layer, a basic component layer, a core implementation layer, a command layer, a command interception layer, a business interface layer and a process engine; Determine a packaging method corresponding to the multiple levels of the layered architecture; wherein the packaging method is bottom-up packaging.
9. A device for implementing a scientific workflow engine, characterized in that: A system for implementing a scientific workflow engine is provided, wherein the system comprises a data management module, a scientific computing module and a workflow management module, and the device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the implementation method of a scientific workflow engine as described in any one of claims 1-8.
10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: A system for implementing a scientific workflow engine is applied, the system includes a data management module, a scientific computing module and a workflow management module, and when the computer executable instructions are executed, a method for implementing a scientific workflow engine as described in any one of claims 1 to 8 is implemented.
