A data quality checking system and method for a dolphin-scheduled data warehouse

Abstract

The application provides a data quality checking system of a data warehouse based on dolphin scheduling, comprising a data quality visualization editing module, a workflow generation module and a dolphin scheduling module; the data quality visualization editing module comprises a data source configuration module and a data quality definition module; the data source configuration module is used for connecting different data sources and obtaining data to be checked; the data quality definition module is used for creating a data quality checking script, configuring a data quality checking rule, and generating a Json format file based on the created data quality checking script; the workflow generation module is used for generating a workflow; the application generates a Json format file based on the created data quality checking script, generates a workflow through a workflow generation template, and automatically checks the integrity, accuracy and consistency of data after the dolphin scheduling module schedules a task to be executed.

Description

Technical Field

[0001] This invention relates to the field of data quality verification technology, and in particular to a data quality verification system and method for a data warehouse based on dolphin scheduling. Background Technology

[0002] Currently, data warehouses are built on open-source Hadoop + Hive (a data warehouse tool based on Hadoop) to construct big data data warehouses. DolphinScheduler is used to schedule data warehouse tasks on a timer. After the data warehouse scheduling execution is completed, the integrity, timeliness, accuracy, and consistency of the loaded target table data cannot be determined. Summary of the Invention

[0003] This invention provides a data quality verification system for data warehouses based on dolphin scheduling, in order to solve the technical problem in the prior art that the integrity, timeliness, accuracy and consistency of the loaded data cannot be determined after the data warehouse scheduling execution is completed.

[0004] One aspect of the present invention is to provide a data quality verification system for a data warehouse based on dolphin scheduling, the system comprising: a data quality visualization and editing module, a workflow generation module, and a dolphin scheduling module;

[0005] The data quality visualization and editing module includes a data source configuration module and a data quality definition module; the data source configuration module is used to configure the connection information of databases from different data sources, connect to different data sources, and obtain the data to be verified.

[0006] The data quality definition module is used to create data quality verification scripts, configure data quality verification rules, and generate JSON format files of the data to be verified based on the created data quality verification scripts.

[0007] The workflow generation module is used to generate workflows; the dolphin scheduling module is used to receive the workflows generated by the workflow generation module.

[0008] The workflow generation module includes a shell task template and a shell workflow template, wherein the shell task template includes multiple first attributes.

[0009] At least one execution script is included among the plurality of the first attributes, wherein the attribute value of the execution script is configured as a variable;

[0010] The shell workflow template includes multiple second attributes, and at least one shell task is included in the multiple second attributes, wherein the attribute value of the shell task is configured as a variable.

[0011] In some preferred embodiments, the shell task template is used to replace the attribute values ​​of the executed script with a JSON format file generated from the data to be verified passed to the workflow generation module, thereby generating a shell task file.

[0012] In some preferred embodiments, the workflow generation module generates multiple shell task files using the shell task template.

[0013] In some preferred embodiments, a shell workflow template is used to replace the attribute values ​​of a shell task with a shell task file generated by the shell task template to generate a workflow.

[0014] In some preferred embodiments, the data quality visualization and editing module transmits the JSON format file generated from the data to be verified to the workflow generation module by calling the application programming interface of the workflow generation module and performing state transitions in JSON format.

[0015] In some preferred embodiments, the workflow generation module sends the generated workflow to the dolphin scheduling module by calling the application programming interface of the dolphin scheduling module.

[0016] In some preferred embodiments, the data quality visualization and editing module further includes a data quality anomaly display module and a data quality task log module;

[0017] The data quality anomaly display module is used to display data quality verification results that do not meet expectations;

[0018] The data quality task log module is used to view the data quality tasks that have been run.

[0019] Another aspect of the present invention is to provide a data quality verification method for a data warehouse based on dolphin scheduling, the method comprising the following steps:

[0020] S101. Configure data source connection information;

[0021] S102. Create a data quality verification script;

[0022] S103, Configure data quality verification rules;

[0023] S104. Configure the verification content of the data quality verification rules;

[0024] S105. Configure the preset conditions for data quality verification results;

[0025] S106. Configure the triggering conditions for the data quality definition module;

[0026] S107. The data to be verified is transmitted to the workflow generation module in JSON format;

[0027] S108. Replace the attribute value of the first attribute of the shell task template in the workflow generation module to generate a shell task file;

[0028] S109. Replace the attribute value of the second attribute of the shell workflow template in the workflow generation module, and generate the workflow;

[0029] S110. The generated workflow is fed into the Dolphin scheduling module, where the data quality of the data to be verified under the workflow is verified.

[0030] In some preferred embodiments, in step S108, the attribute values ​​of the execution script of the shell task template in the workflow generation module are replaced with a JSON format file generated from the data to be verified passed to the workflow generation module, thereby generating a shell task file.

[0031] In some preferred embodiments, the attribute values ​​of the shell task in the shell workflow template in the workflow generation module are replaced with the shell task file generated by the shell task template in the workflow generation module to generate the workflow.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] This invention provides a data quality verification system and method for a data warehouse based on Dolphin scheduling. The system generates a JSON file from the data to be verified based on a created data quality verification script, generates a workflow through a workflow generation template, and automatically verifies the integrity, accuracy, and consistency of the data after the Dolphin scheduling module completes the task execution. Attached Figure Description

[0034] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 This is a structural block diagram of a data quality verification system for a data warehouse based on dolphin scheduling, according to the present invention.

[0036] Figure 2 This is a schematic diagram of a shell task template in one embodiment of the present invention.

[0037] Figure 3 This is a schematic diagram of a shell workflow template in one embodiment of the present invention.

[0038] Figure 4 This is a flowchart of a data quality verification method for a data warehouse based on dolphin scheduling according to the present invention. Detailed Implementation

[0039] To make the above and other features and advantages of the present invention clearer, the invention will be further described below with reference to the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explanation to those skilled in the art and are exemplary only, not restrictive.

[0040] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0041] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0042] like Figure 1 The diagram shown is a structural block diagram of a data quality verification system for a data warehouse based on dolphin scheduling according to the present invention. According to an embodiment of the present invention, a data quality verification system for a data warehouse based on dolphin scheduling is provided, including: a data quality visualization and editing module 100, a workflow generation module 200, and a dolphin scheduling module 300.

[0043] The data quality visualization and editing module 100 is built using the Vue front-end framework and includes a data source configuration module 101, a data quality anomaly display module 102, a data quality definition module 103, and a data quality task log module 104.

[0044] The data quality visualization and editing module 100 encapsulates the data source configuration module 101, the data quality anomaly display module 102, the data quality definition module 103, and the data quality task log module 104 into a user-operable UI interface.

[0045] The data source configuration module 101 is used to configure the connection information of databases from different data sources, connect to different data sources, and obtain the data to be verified.

[0046] Different data sources include, but are not limited to, MySQL, Hive / Impala, Spark, ClickHouse, and Oracle databases.

[0047] In one embodiment, the data source configuration module 101 configures the connection information for databases from different data sources, as follows:

[0048] Data source name: Creates a unique identifier for the data source name.

[0049] Description: Records detailed information about the data source.

[0050] Host address: The address for the database connection.

[0051] Port: Database connection port.

[0052] Username: The username used to connect to the database.

[0053] Password: Database connection password.

[0054] Database name: The specific name of the database to connect to.

[0055] JDBC connection parameters (Java Database Connectivity, or JDBC for short): Configure less commonly used database connection parameters.

[0056] JSON (JavaScript Object Notation) format parameters: added via key-value pairs.

[0057] The data quality anomaly display module 102 is used to display data quality verification results that do not meet expectations.

[0058] In one embodiment, the information displayed by the data quality anomaly display module 102 is as follows:

[0059] Project: Which specific project has the data anomaly?

[0060] Workflow: Data anomalies within a specific workflow.

[0061] Task Name: The specific task name within a workflow.

[0062] Start Time: The start time of data quality verification, accurate to the year, month, day, hour, minute, and second.

[0063] End Time: The end time of data quality verification, accurate to the year, month, day, hour, minute, and second.

[0064] Verification status: There are four specific statuses for data quality tasks, such as: not verified, verification in progress, failed, and passed.

[0065] Maintainer: The name of the person responsible for maintaining the data quality task.

[0066] Action: Take action to address data quality anomalies.

[0067] Ignore: Handle data quality anomalies normally.

[0068] Rerun: Perform the data quality verification rules again.

[0069] View logs: View the specific execution logs of data quality verification.

[0070] The data quality definition module 103 is used to create data quality verification scripts, configure data quality verification rules, and generate JSON format files of the data to be verified based on the created data quality verification scripts.

[0071] According to an embodiment of the present invention, the data quality definition module 103 is an interface for operating data quality information, specifically including operations such as creation, query, deletion, modification, offline, timed, and copying.

[0072] The data quality definition module 103 of this invention creates a data quality verification script, configures data quality verification rules, and generates a JSON format file of the data to be verified based on the created data quality verification script.

[0073] The data quality definition module 103 creates data quality verification script information including workflow name, task name, quality task name, startup status, calculation rules, data source type, data source name, database name, table name, field name, date field, date dynamic parameter, data retrieval rules, data quality verification rules, verification content of data quality verification rules, preset conditions for data quality verification results, trigger conditions for the data quality definition module, and stopping the current workflow if the task fails.

[0074] The data quality verification script created in the data quality definition module 103 configures the data quality verification rules, the verification content of the data quality verification rules, the preset conditions for the data quality verification results, and the trigger conditions for the data quality definition module.

[0075] The data source configuration module 101 configures the connection information of databases from different data sources and obtains the data to be verified. The data quality definition module 103 generates a JSON file from the data to be verified based on the created data quality verification script.

[0076] In the data quality verification script created by the data quality definition module 103, the data retrieval rule is to specify the template for data quality verification. The template information includes: number of records, summary value, number of duplicate records, and custom information.

[0077] In the embodiments of the present invention, the verification content of the data quality verification rule is the specific verification SQL (Structured Query Language) logic, and the execution result of the verification SQL logic is divided into: dimension fields, record number fields, and summary fields.

[0078] In the data quality verification script created by the data quality definition module 103, the date field: obtains date restrictions based on the selected data to be verified. The dynamic date parameter: dynamically verifies the data to be verified for a specific day based on the selected date field, such as yesterday.

[0079] The data quality definition module 103 of this invention also includes operations such as querying, deleting, modifying, taking offline, scheduling, and copying data quality information, specifically:

[0080] Query: Used to query existing data quality verification rules.

[0081] Delete: Used to delete data quality verification rules that have already been created.

[0082] Edit: Used to modify existing data quality verification rules.

[0083] Take offline: Used to take offline data quality verification rules that have been created and put online.

[0084] Scheduled: Used to specify data quality verification rules that have already been created at regular intervals.

[0085] Copy: Used to copy created data quality verification rules.

[0086] The data quality task log module 104 is used to view the data quality tasks that have been running. Specific data quality task information includes: workflow name, task name, start execution time, end execution time, running status, verification status, maintainer, stop, rerun, and view log.

[0087] According to an embodiment of the present invention, the workflow generation module 200 runs on a server, and the data quality visualization and editing module 100 calls the workflow generation module 200 through a REST (Resource Representational State Transfer) API (Application Programming Interface). The data quality visualization and editing module 100, based on a created data quality verification script, transmits the data to be verified as a JSON file to the workflow generation module 200.

[0088] Workflow generation module 200 is used to generate workflows. Dolphin scheduling module 300 is used to receive the workflow generated by workflow generation module 200.

[0089] The workflow generation module 200 includes a shell task template 201 and a shell workflow template 202. The shell task template 201 includes multiple first attributes in JSON format. For example, the multiple first attributes are attribute a, attribute b, attribute c, ...

[0090] According to an embodiment of the present invention, at least one of the multiple first attributes of the shell task template 201 includes an execution script, wherein the attribute value of the execution script is configured as a variable. The shell task template 201 is used to replace the attribute value of the execution script with a JSON format file generated from the data to be verified passed to the workflow generation module 200, thereby generating a shell task file.

[0091] Figure 2 This is a schematic diagram of a shell task template in one embodiment of the present invention. For example, among the multiple first attributes, attribute a is the execution script, attribute b is the task name (which is consistent with the name of the JSON format file generated from the data to be verified). In some embodiments, the first attributes also include task code, task type (Shell), custom parameters, task description (default: automatically generated), number of retries (default: 2), retry interval (default: 2), ...

[0092] According to an embodiment of the present invention, among the multiple first attributes, the attribute value of the execution script (corresponding to a) is configured as a variable, for example, the variable $execScript. A shell task file is generated by replacing the attribute value of the execution script (corresponding to a) of the shell task template 201 with a JSON format file generated from the data to be verified passed to the workflow generation module 200.

[0093] This invention generates the final executable shell task file of the Dolphin Scheduling Module 300 by replacing the attribute values ​​of the execution script (corresponding to a) of the shell task template 201.

[0094] According to an embodiment of the present invention, the shell workflow template 202 includes a plurality of second attributes. For example, the plurality of second attributes are attribute A, attribute B, attribute C, ...

[0095] According to an embodiment of the present invention, at least one shell task is included among the plurality of second attributes of the shell workflow template 202, wherein the attribute value of the shell task is configured as a variable. The shell workflow template 202 is used to replace the attribute value of the shell task with the shell task file generated by the shell task template 201 to generate a workflow.

[0096] like Figure 3 The diagram shown illustrates a shell workflow template in one embodiment of the present invention. Exemplarily, among multiple second attributes, attribute A is the Shell task, and attribute B is the workflow name (consistent with the Shell task name). In some embodiments, the second attributes further include a workflow description (default: automatically generated), creation time, modification time, scheduled time, etc.

[0097] According to an embodiment of the present invention, among a plurality of second attributes, the attribute value of the Shell task (corresponding to belonging to A) is configured as a variable. A workflow is generated by replacing the attribute value of the Shell task (corresponding to belonging to A) in the shell workflow template 202 with the shell task file generated by the shell task template 201.

[0098] A workflow is the smallest executable task in Dolphin scheduling, and a workflow can contain multiple shell tasks. In one embodiment, the workflow generation module 200 generates multiple shell task files using shell task template 201.

[0099] In this invention, each shell task file in the workflow is formed by replacing the attribute values ​​of the execution script (corresponding to 'a') in shell task template 201 with a JSON format file generated from the data to be verified by the workflow generation module 200. Each shell task file also replaces the attribute values ​​of the Shell task (corresponding to 'A') in shell workflow template 202, forming the final executable workflow of the Dolphin Scheduling Module 300 encapsulated in JSON format.

[0100] According to an embodiment of the present invention, the data quality visualization and editing module 100 transmits the JSON format file generated from the data to be verified to the workflow generation module 200 by calling the application programming interface of the workflow generation module 200 in a JSON format state transition manner.

[0101] In this embodiment of the invention, the workflow generation module 200 uses a Rest (Resource Representational State Transfer) API (Application Programming Interface) to call the workflow generation module 200's application programming interface and pass the JSON format file generated from the data to be verified into the workflow generation module 200.

[0102] According to an embodiment of the present invention, the workflow generation module 200 inputs the generated workflow into the dolphin scheduling module 300 by calling the application programming interface of the dolphin scheduling module 300.

[0103] In an embodiment of the present invention, the Dolphin scheduling module 300 runs on a server. The application programming interface of the Dolphin scheduling module 300 adopts an API (Application Programming Interface). The workflow generation module 200 calls the application programming interface of the Dolphin scheduling module 300 to send the generated workflow into the Dolphin scheduling module 300 and store it in metadata.

[0104] Metadata, also known as intermediary data or relay data, mainly describes information about data properties and is used to support functions such as indicating storage location, historical data, resource lookup, and file records. The metadata of the Dolphin Scheduling Module 300 in this invention is used to store the executable workflows of the Dolphin Scheduling Module 300.

[0105] The Dolphin Scheduling Module 300 of this invention performs data quality verification on the running workflow and the tasks (data to be verified) under the running workflow, and stores the running results (data quality verification results) of the workflow and the tasks (data to be verified) under the workflow in the metadata.

[0106] like Figure 4 The flowchart shown is a data quality verification method for a data warehouse based on dolphin scheduling according to the present invention. According to an embodiment of the present invention, a data quality verification method for a data warehouse based on dolphin scheduling is provided, including the following method steps:

[0107] Step S101: Configure data source connection information.

[0108] Configure the database connection information for different data sources in the data source configuration module 101, connect to different data sources, and obtain the data to be verified.

[0109] Step S102: Create a data quality verification script.

[0110] Create a data quality verification script in the data quality definition module 103.

[0111] Step S103: Configure data quality verification rules.

[0112] Step S104: Configure the verification content of the data quality verification rules.

[0113] Step S105: Configure the preset conditions for data quality verification results.

[0114] Step S106: Configure the triggering conditions for the data quality definition module.

[0115] In steps S103 to S106, the data quality verification script created by the data quality definition module 103 is configured with data quality verification rules, verification content of the data quality verification rules, preset conditions for data quality verification results, and trigger conditions for the data quality definition module.

[0116] Data quality verification rules are used to determine whether the data verification results meet expectations.

[0117] The data quality verification rules verify specific SQL (Structured Query Language) logic. The execution results of the verification SQL logic are divided into: dimension fields, record numeric fields, and summary fields.

[0118] In one embodiment, when the verification content of the data quality verification rule is filled with the number of verification records, the preset condition for the data quality verification result is configured as the number of records not being less than 0.

[0119] The triggering condition for the data quality definition module 103 is the condition for its startup and operation. In some embodiments, the data quality definition module 103 is triggered to start and operate at regular intervals; in other embodiments, the data quality definition module 103 is triggered to start and operate based on task events, that is, it is triggered to start and operate when data verification is required.

[0120] Step S107: The data to be verified is transmitted to the workflow generation module in JSON format.

[0121] The data quality definition module 103 creates a data quality verification script, generates a JSON file of the data to be verified based on the created data quality verification script, and passes the data to be verified into the workflow generation module 200 in JSON format.

[0122] Step S108: Replace the attribute value of the first attribute of the shell task template in the workflow generation module to generate a shell task file.

[0123] According to an embodiment of the present invention, the attribute values ​​of the execution script of the shell task template 201 in the workflow generation module 200 are replaced with a JSON format file generated from the data to be verified passed into the workflow generation module 200, thereby generating a shell task file.

[0124] For example, the attribute values ​​of the execution script in the shell task template 201 of the workflow generation module 200 before replacement were:

[0125] "params":{

[0126] "rawScript":"$execScript"

[0127] }

[0128] The attribute values ​​of the execution script in the shell task template 201 of the workflow generation module 200, after being replaced with the JSON format file generated from the data to be verified passed to the workflow generation module 200, are as follows:

[0129] "params":{

[0130] "rawScript":"echo"This is a JSON file generated from the data to be validated."

[0131] }

[0132] A workflow is the smallest executable task in Dolphin scheduling, and a workflow can contain multiple shell tasks. In one embodiment, the workflow generation module 200 generates multiple shell task files using shell task template 201.

[0133] Step S109: Replace the attribute value of the second attribute of the shell workflow template in the workflow generation module to generate the workflow.

[0134] According to an embodiment of the present invention, the attribute values ​​of the shell task in the shell workflow template 202 in the workflow generation module 200 are replaced with the shell task file generated by the shell task template 201 in the workflow generation module 200 to generate a workflow.

[0135] For example, the attribute values ​​of the shell task in shell workflow template 202 in workflow generation module 200 are replaced with the shell task file generated by shell task template 201 in workflow generation module 200. The resulting workflow will ultimately form a workflow in the following JSON format: {

[0136] "projectName":"Data Warehouse",

[0137] "processDefinitionName":"Workflow Name",

[0138] "processDefinitionJson":"Shell task (This is a shell task file generated from a shell task template 201)",

[0139] "processDefinitionDescription":"Data Integration",

[0140] "processDefinitionLocations":"Workflow location information",

[0141] "processDefinitionConnects":"[]"

[0142] }

[0143] S110. The generated workflow is fed into the Dolphin scheduling module, where the data quality of the data to be verified under the workflow is verified.

[0144] The workflow generation module 200 calls the application programming interface of the Dolphin scheduling module 300 to send the JSON format file generated from the data to be verified into the Dolphin scheduling module 300 as a workflow encapsulated in JSON format. The Dolphin scheduling module 300 runs the workflow and the tasks (data to be verified) under the workflow to perform data quality verification, and stores the running results (data quality verification results) of the workflow and the tasks (data to be verified) under the workflow in the metadata.

[0145] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A data quality verification system for a data warehouse based on dolphin scheduling, characterized in that, The system includes: a data quality visualization and editing module, a workflow generation module, and a dolphin scheduling module; The data quality visualization and editing module includes a data source configuration module and a data quality definition module; the data source configuration module is used to configure the connection information of databases from different data sources, connect to different data sources, and obtain the data to be verified. The data quality definition module is used to create data quality verification scripts, configure data quality verification rules, and generate JSON format files of the data to be verified based on the created data quality verification scripts. The workflow generation module is used to generate workflows; the dolphin scheduling module is used to receive the workflows generated by the workflow generation module. The workflow generation module includes a shell task template and a shell workflow template, wherein the shell task template includes multiple first attributes. At least one execution script is included among the plurality of the first attributes, wherein the attribute value of the execution script is configured as a variable; The shell task template is used to replace the attribute values ​​of the executed script with a JSON format file generated from the data to be verified passed to the workflow generation module, thereby generating multiple shell task files. The shell workflow template includes multiple second attributes, and at least one shell task is included in the multiple second attributes, wherein the attribute value of the shell task is configured as a variable; The shell workflow template is used to replace the attribute values ​​of the shell task with the shell task file generated by the shell task template to generate a workflow; The Dolphin scheduling module performs data quality checks on the workflow and the tasks under the workflow, and stores the results of the workflow and the tasks under the workflow in the metadata.

2. The system according to claim 1, characterized in that, The data quality visualization and editing module transmits the generated JSON file of the data to be verified to the workflow generation module by calling the application programming interface of the workflow generation module and performing state transitions in JSON format.

3. The system according to claim 1, characterized in that, The workflow generation module sends the generated workflow to the Dolphin scheduling module by calling the application programming interface of the Dolphin scheduling module.

4. The system according to claim 1, characterized in that, The data quality visualization and editing module also includes a data quality anomaly display module and a data quality task log module; The data quality anomaly display module is used to display data quality verification results that do not meet expectations; The data quality task log module is used to view the data quality tasks that have been run.

5. A method for data quality verification using the data quality verification system for a data warehouse based on dolphin scheduling as described in any one of claims 1 to 4, characterized in that, The method includes the following steps: S101. Configure data source connection information; S102. Create a data quality verification script; S103, Configure data quality verification rules; S104. Configure the verification content of the data quality verification rules; S105. Configure the preset conditions for data quality verification results; S106. Configure the triggering conditions for the data quality definition module; S107. The data to be verified is transmitted to the workflow generation module in JSON format; S108. Replace the attribute value of the first attribute of the shell task template in the workflow generation module to generate a shell task file; S109. Replace the attribute value of the second attribute of the shell workflow template in the workflow generation module, and generate the workflow; S110. The generated workflow is fed into the Dolphin scheduling module, where the data quality of the data to be verified under the workflow is verified.

6. The method according to claim 5, characterized in that, In step S108, the attribute values ​​of the execution script of the shell task template in the workflow generation module are replaced with a JSON format file generated from the data to be verified passed to the workflow generation module, thereby generating a shell task file.

7. The method according to claim 6, characterized in that, The attribute values ​​of the shell task in the shell workflow template in the workflow generation module are replaced with the shell task file generated by the shell task template in the workflow generation module to generate the workflow.

Images