Workflow creation method, device and equipment of data warehouse and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a data warehouse workflow creation method and device, equipment and a storage medium, wherein the method comprises the following steps: taking any one of a set of business system identifiers as a target business system identifier; executing a workflow creation script, generating a target workflow name and each target node name according to the target business system identifier, workflow table information data and data processing service configuration data; creating workflow description information in a target XML file according to the target workflow name, each target node name and the data processing service configuration data; repeating the step of determining the target business system identifier until the set of business system identifiers is acquired; and taking the target XML file as a workflow creation result. The method realizes automatic batch workflow creation in an existing XML file, and avoids manual creation of nodes and configuration of node relationships.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for creating workflows for a data warehouse. Background Technology

[0002] Existing big data platforms suffer from the inability to batch upload nodes (corresponding to a single task) and workflows. Nodes must be manually created and their dependencies configured to deploy a workflow. Information about scheduling nodes (node ​​names and dependencies) is stored in XML (Extensible Markup Language) files, and this information needs to be generated based on the node type. Furthermore, all workflows within a project space reside in the same XML file. Therefore, creating nodes and configuring dependencies via XML requires manual creation and configuration within existing XML files. When there are many or complex nodes, manual operation leads to error-prone XML modifications and inefficient workflow creation. Summary of the Invention

[0003] The main purpose of this application is to provide a workflow creation method, apparatus, device, and storage medium for a data warehouse, aiming to solve the technical problems of existing big data platforms, such as the inability to upload nodes and workflows in batches, and the fact that all workflows under a project space are in the same XML file, which means that nodes can only be created and dependency relationships configured manually in the existing XML file, resulting in errors in XML modification and low workflow creation efficiency.

[0004] To achieve the above-mentioned objectives, this application proposes a workflow creation method for a data warehouse, the method comprising:

[0005] Obtain a workflow creation request, wherein the workflow creation request carries a business system identifier set;

[0006] In response to the workflow creation request, obtain workflow table information data, data processing service configuration data, workflow creation script, and target XML file;

[0007] Use any one of the business system identifiers in the set of business system identifiers as the target business system identifier;

[0008] The workflow creation script is executed to generate the target workflow name and the names of each target node based on the target business system identifier, the workflow table information data, and the data processing service configuration data.

[0009] executing the workflow creation script, and creating workflow description information in the target XML file according to the target workflow name, each target node name and the data processing service configuration data;

[0010] The step of taking any one of the business system identifiers in the set of business system identifiers as a target business system identifier is repeated until the acquisition of the business system identifiers in the set of business system identifiers is completed, and the target XML file is taken as a workflow creation result.

[0011] Further, the step of generating a target workflow name and each target node name according to the target business system identifier, the workflow table information data and the data processing service configuration data respectively includes:

[0012] A preset workflow naming specification is adopted to generate a workflow name according to the target business system identifier, the workflow table information data and the data processing service configuration data, and the target workflow name is obtained.

[0013] A workflow template is acquired, a preset node naming specification is adopted, and node names are generated according to the workflow template, the target business system identifier, the workflow table information data and the data processing service configuration data, and each target node name is obtained.

[0014] Further, the target business system identifier corresponds to one target workflow name, and the target business system identifier corresponds to a target node name of a start node and each target node name of each process branch.

[0015] The process branch corresponds to a data table of a business system corresponding to the target business system identifier in a one-to-one manner, and each process branch includes a synchronous data buffer layer node, a synchronous data source layer node and a synchronous data detail layer node.

[0016] Further, the step of creating workflow description information in the target XML file according to the target workflow name, each target node name and the data processing service configuration data includes:

[0017] In the target XML file, fixed tags of the workflow description information are created according to a workflow XML generation rule, wherein the fixed tags include a compression type tag, a solution tag and an internal file tag.

[0018] Below the fixed tag in the target XML file, basic information and flow information of the workflow description information are created according to the target workflow name, each target node name and the data processing service configuration data.

[0019] Further, the step of creating the basic information and flow information of the workflow description information in the target XML file according to the target workflow name, each target node name and the data processing service configuration data comprises:

[0020] In the target XML file, a workflow basic information tag and a workflow specific information tag are created according to the target workflow name and the data processing service configuration data respectively.

[0021] In the target XML file, a workflow node information tag is created and node specific information and node dependency relationship information in the workflow node information tag are created according to the target workflow name, each target node name and the data processing service configuration data respectively.

[0022] Further, the step of creating the workflow node information tag and the node specific information and node dependency relationship information in the workflow node information tag in the target XML file according to the target workflow name, each target node name and the data processing service configuration data respectively comprises:

[0023] In the target XML file, the creation of the workflow node information tag is performed by using a preset tag creation rule.

[0024] In the workflow node information tag in the target XML file, a node sub-tag is created for each target node name.

[0025] In the node sub-tag in the target XML file, a node specific information sub-tag, a node dependency relationship information sub-tag and a current node input / output information sub-tag are created according to each target node name and the workflow node information tag.

[0026] In the target XML file, attribute information of the node sub-tag is set according to each target node name and the data processing service configuration data.

[0027] In the target XML file, the node specific information sub-tag, the node dependency relationship information sub-tag and the current node input / output information sub-tag are set according to the data processing service configuration data.

[0028] Further, the step of creating fixed tags of the workflow description information in the target XML file according to the workflow XML generation rule comprises:

[0029] In the target XML file, a compression type tag, a solution tag and an internal file tag are created according to the workflow XML generation rule.

[0030] The application further provides a workflow creation device of a data warehouse, and the device comprises:

[0031] A request acquisition module is configured to acquire a workflow creation request, wherein the workflow creation request carries a business system identifier set.

[0032] A data acquisition module is configured to acquire workflow table information data, data processing service configuration data, a workflow creation script and a target XML file in response to the workflow creation request.

[0033] A target business system identifier determination module is configured to determine any business system identifier in the business system identifier set as a target business system identifier.

[0034] A workflow name and node name generation module is configured to execute the workflow creation script, and generate a target workflow name and each target node name according to the target business system identifier, the workflow table information data and the data processing service configuration data.

[0035] A workflow description information creation module is configured to execute the workflow creation script, and create workflow description information in the target XML file according to the target workflow name, each target node name and the data processing service configuration data.

[0036] A loop control module is configured to repeatedly execute the step of determining any business system identifier in the business system identifier set as a target business system identifier until the acquisition of the business system identifiers in the business system identifier set is completed, and to take the target XML file as a workflow creation result.

[0037] The application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0038] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program implements the steps of any of the above methods when executed by a processor.

[0039] The workflow creation method, device, equipment and storage medium of the data warehouse of the application, wherein the method executes the workflow creation script by taking any one of the business system identifiers in the set of business system identifiers as a target business system identifier, generates a target workflow name and each target node name according to the target business system identifier, the workflow table information data and the data processing service configuration data respectively, executes the workflow creation script, creates workflow description information in the target XML file according to the target workflow name, each target node name and the data processing service configuration data, repeatedly executes the step of taking any one of the business system identifiers in the set of business system identifiers as a target business system identifier until the acquisition of the business system identifiers in the set of business system identifiers is completed, and takes the target XML file as a workflow creation result, thereby realizing automatic batch workflow creation in the existing XML file, avoiding manual creation of nodes and configuration of dependent node relationships, improving the creation efficiency of the workflow and improving the accuracy of the workflow. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A flowchart of a workflow creation method of a data warehouse according to an embodiment of the application;

[0041] Figure 2 A structural schematic block diagram of a workflow creation device of a data warehouse according to an embodiment of the application;

[0042] Figure 3 A structural schematic block diagram of a computer device according to an embodiment of the application.

[0043] The implementation, functional features and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the application more clear, the application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application.

[0045] Referring to Figure 1 A workflow creation method of a data warehouse is provided in the embodiments of the application, and the method comprises:

[0046] S1: Acquire a workflow creation request, wherein the workflow creation request carries a set of business system identifiers;

[0047] S2: In response to the workflow creation request, acquire workflow table information data, data processing service configuration data, a workflow creation script and a target XML file;

[0048] S3: taking any one of the business system identifiers in the set of business system identifiers as a target business system identifier;

[0049] S4: executing the workflow creation script to generate a target workflow name and each target node name according to the target business system identifier, the workflow table information data, and the data processing service configuration data, respectively;

[0050] S5: executing the workflow creation script to create workflow description information in the target XML file according to the target workflow name, each target node name, and the data processing service configuration data;

[0051] S6: repeating the step of taking any one of the business system identifiers in the set of business system identifiers as a target business system identifier until the acquisition of the business system identifiers in the set of business system identifiers is completed, and taking the target XML file as a workflow creation result.

[0052] The embodiment takes any one of the business system identifiers in the set of business system identifiers as a target business system identifier, executes the workflow creation script to generate a target workflow name and each target node name according to the target business system identifier, the workflow table information data, and the data processing service configuration data, respectively, executes the workflow creation script to create workflow description information in the target XML file according to the target workflow name, each target node name, and the data processing service configuration data, and repeats the step of taking any one of the business system identifiers in the set of business system identifiers as a target business system identifier until the acquisition of the business system identifiers in the set of business system identifiers is completed, and takes the target XML file as a workflow creation result, thereby realizing the automatic batch workflow creation in the existing XML file, avoiding the manual creation of nodes and the configuration of dependent node relationships, improving the creation efficiency of the workflow, and improving the accuracy of the workflow.

[0053] For S1, the workflow creation request input by the user or the workflow creation request sent by a third-party application system can be acquired.

[0054] The workflow creation request is a request for creating workflow description information in the XML file. The workflow description information is the description information of the nodes and the dependent node relationships.

[0055] The set of business system identifiers includes one or more business system identifiers. The business system identifier can be a business system name, a business system ID, or other data uniquely identifying a business system. The business system is an application system providing business services.

[0056] The workflow of the present application is used to synchronize the data of each data table of a business system into a data warehouse. For example, the workflow includes a start node, an SRC2STG node (i.e., a synchronization data buffer layer node), an STG2ODS node (i.e., a synchronization data source layer node), and an ODS2DWD node (i.e., a synchronization data detail layer node). The start node indicates that the workflow starts to run and has no other special meaning. The SRC2STG node is used to extract data from the data table of the business system to the data buffer layer (STG layer) of the data warehouse. The STG2ODS node is used to extract data from the STG layer to the data source layer (ODS layer) of the data warehouse. The ODS2DWD node is used to extract data from the ODS layer to the data detail layer (DWD layer) of the data warehouse. The above is an example and is not limited in particular.

[0057] Each node corresponds to a task.

[0058] It can be understood that in the present application, each workflow synchronizes the data of each data table corresponding to a business system into a data warehouse.

[0059] It can be understood that in another embodiment of the present application, each workflow synchronizes the data of a data table corresponding to a business system into a data warehouse. That is, each workflow corresponds to a data table.

[0060] For S2, when the workflow creation request is received, the workflow table information data, the data processing service configuration data, the workflow creation script, and the target XML file can be obtained from the database, or the workflow table information data, the data processing service configuration data, the workflow creation script, and the target XML file input by the user can be obtained, or the workflow table information data, the data processing service configuration data, the workflow creation script, and the target XML file can be obtained from a third-party application system.

[0061] The workflow table information data is description information used to describe the data table to be processed by the workflow. The workflow table information data includes but is not limited to the table name of the data table, the DB name to which the data table belongs, the system code abbreviation to which the data table belongs, and the description information of the table name. The table name of the data table is the name of the data table of the business system. The DB name to which the data table belongs is the database name to which the data table belongs. The system code abbreviation to which the data table belongs is the code abbreviation of the business system to which the data table belongs. The description information of the table name is the description information of the data table of the business system.

[0062] The data processing service configuration data is configuration data formed by parameters such as the project space name of Dataworks, the creator account, and the workflow life cycle.

[0063] Dataworks, a one-stop development workshop based on the MaxCompute (data warehouse) computing engine, helps enterprises quickly complete data integration, development, governance, quality, security, and other complete data research and development work. The workflow created on Dataworks, the basic information of the workflow (workflow name, workflow project space, workflow owner), and process information (sub-nodes, dependencies between sub-nodes) are stored in an XML file.

[0064] The target XML file is an XML format file that stores workflow description information. Workflow description information is the information of the scheduling nodes, including the names of each node and the dependencies between nodes.

[0065] XML is the Extensible Markup Language, which is a subset of the Standard Generalized Markup Language.

[0066] The workflow creation script is a script developed in Java (an object-oriented programming language) and is used to create workflow description information in an XML file. It should be understood that the workflow creation script can also be developed in other languages, such as Python (a computer programming language), without limitation.

[0067] When the workflow creation script is developed in Java, the Dom4j package is referenced, each tag and sub-tag is created through addElement, and the attributes of the tag and sub-tag are created through AddAttribute. The Dom4j package is an XML API for Java, which is an upgrade of Jdom and is used to read and write XML files. addElement is a special method in the Vector class, and the return value is void.

[0068] For S3, any business system identifier is obtained from the business system identifier set, and the obtained business system identifier is used as the target business system identifier.

[0069] For S4, the workflow creation script is executed, and a preset naming specification is used to generate a target workflow name and each target node name according to the target business system identifier, the workflow table information data, and the data processing service configuration data.

[0070] The preset naming specification includes a preset workflow naming specification and a preset node naming specification.

[0071] The preset workflow naming specification is expressed as: "prefix" + system code abbreviation + DB name. The prefix is used to identify the type as a workflow. For example, the prefix is the abbreviation of the English name of the workflow (workflow), which is wf, and this example is not limited in detail.

[0072] The preset node naming specification is expressed as "node purpose abbreviation" + system code abbreviation + DB name + table name. The node purpose abbreviation is used to express the purpose of the node. For example, the node purpose abbreviations are SRC2STG, STG2ODS and ODS2DWD, which are not limited in the examples.

[0073] For S5, if the node and the dependent node relationship are created by means of XML, the node and the dependent node relationship can only be created in the existing XML file. Therefore, the workflow creation script is executed, the workflow description information is created in the target XML file based on the workflow XML generation rule according to the target workflow name, each target node name and the data processing service configuration data, so that the creation of the workflow corresponding to the target business system identifier is realized.

[0074] The workflow XML generation rule is a rule obtained by parsing the XML format workflow description information file.

[0075] For S6, steps S3 to S6 are repeatedly executed until the acquisition of the business system identifiers in the business system identifier set is completed. When the acquisition of the business system identifiers in the business system identifier set is completed, the batch creation of the workflows in the target XML file is completed, and therefore the target XML file can be used as the workflow creation result.

[0076] In one embodiment, the above-mentioned step of generating the target workflow name and each target node name according to the target business system identifier, the workflow table information data and the data processing service configuration data respectively includes:

[0077] S41: generating the workflow name according to the target business system identifier, the workflow table information data and the data processing service configuration data by using the preset workflow naming specification, to obtain the target workflow name;

[0078] S42: obtaining a workflow template, generating the node name according to the workflow template, the target business system identifier, the workflow table information data and the data processing service configuration data by using the preset node naming specification, to obtain each target node name.

[0079] This embodiment realizes the automatic generation of the workflow name by using the preset workflow naming specification and the automatic generation of the node name by using the preset node naming specification and the workflow template, and provides a basis for the automatic creation of the workflow description information.

[0080] For S41, data is acquired from the workflow table information data and the data processing service configuration data according to the target business system identifier, the acquired data is combined into a workflow name according to a preset workflow naming specification, and the combined data is taken as the target workflow name.

[0081] It can be understood that the target business system identifier corresponds to one target workflow name.

[0082] For S42, a workflow template can be acquired from a database or a third-party application system.

[0083] Optionally, a workflow template corresponding to the target business system identifier can be acquired from a database or a third-party application system, so that personalized workflow templates are adopted for different business system identifiers.

[0084] The workflow template includes multiple nodes arranged in a sequence. For example, the workflow template includes, in sequence, a start node, a synchronous data buffer layer node, a synchronous data source layer node, and a synchronous data detail layer node, which are not limited in this example.

[0085] According to the workflow template, data is acquired from the workflow table information data and the data processing service configuration data according to the target business system identifier, each node name is generated according to a preset node naming specification, and each generated node name is taken as one target node name.

[0086] In one embodiment, the target business system identifier corresponds to one target workflow name, and the target business system identifier corresponds to the target node name of a start node and the target node names of each process branch.

[0087] The process branch corresponds to a data table of a business system corresponding to the target business system identifier in a one-to-one manner, and each process branch includes a synchronous data buffer layer node, a synchronous data source layer node, and a synchronous data detail layer node.

[0088] This embodiment creates a workflow for a target business system identifier, and each process branch shares a start node, thereby simplifying workflow creation and facilitating workflow execution control.

[0089] For example, the workflow template comprises, in sequence, a start node, a synchronous data buffer layer node, a synchronous data source layer node, and a synchronous data detail layer node. If the target business system identifier indicates that four data tables need to be synchronized to the data warehouse, the target business system identifier corresponds to a start node target node name, four synchronous data buffer layer nodes, four synchronous data source layer nodes, and four synchronous data detail layer nodes.

[0090] In one embodiment, the step of creating the workflow description information in the target XML file according to the target workflow name, each target node name, and the data processing service configuration data comprises:

[0091] S51: Creating fixed tags of the workflow description information in the target XML file according to the workflow XML generation rule, wherein the fixed tags comprise a compression type tag, a solution tag, and an internal file tag.

[0092] S52: Creating basic information and process information of the workflow description information in the target XML file below the fixed tags according to the target workflow name, each target node name, and the data processing service configuration data.

[0093] This embodiment realizes the automatic creation of fixed tags according to the workflow XML generation rule, and then realizes the automatic creation of basic information and process information of the workflow description information, thereby avoiding manual creation and improving the efficiency and accuracy of creation.

[0094] For S51, the fixed tags are tags that are fixed for each workflow, and therefore can be directly created in the target XML file according to the workflow XML generation rule.

[0095] The fixed tags comprise a compression type tag, a solution tag, and an internal file tag.

[0096] The compression type tag is expressed as: <ziptype>, for expressing the type of data compression by the workflow.

[0097] solution tag, expressed as: <solutions>, for expressing the solution corresponding to the workflow.

[0098] Internal file label, expressed as: <innerfiles>, for expressing files on Dataworks.

[0099] For S52, under the fixed tag in the target XML file, according to the target workflow name, each target node name and the data processing service configuration data, the creation of the workflow tag, the creation of the node tag, the configuration of the node specific information and the node dependency information according to the created node tag are performed, so as to realize the automatic creation of the basic information and the process information of the workflow description information.

[0100] Under the fixed tag in the target XML file, that is, at the tail of the target XML file.

[0101] In one embodiment, the step of creating the basic information and the process information of the workflow description information in the target XML file according to the target workflow name, each target node name and the data processing service configuration data includes:

[0102] S521: In the target XML file, according to the target workflow name and the data processing service configuration data, the creation of the workflow basic information tag and the workflow specific information tag is performed respectively.

[0103] S522: In the target XML file, according to the target workflow name, each target node name and the data processing service configuration data, the creation of the workflow node information tag and the creation of the node specific information and the node dependency information in the workflow node information tag are performed respectively.

[0104] This embodiment first performs the automatic creation of the workflow basic information tag and the workflow specific information tag, and then performs the automatic creation of the workflow node information tag and the automatic creation of the node specific information and the node dependency information in the workflow node information tag, so as to complete the creation of the basic information and the process information of the workflow description information, avoid manual creation, and improve the efficiency and accuracy of the creation.

[0105] For S521, each workflow has a workflow basic information tag, a workflow specific information tag and a workflow node information tag in the XML file.

[0106] The workflow basic information tag is expressed as: <bizinfos>, basic information for describing the workflow, such as a workflow name, an account of a creator of the workflow. There is a workFlowName (that is, a workflow name parameter) in the workflow basic information tag, and thus, when a new workflow is generated, the workFlowName of the workflow basic information tag is replaced with the name of the new workflow.

[0107] A workflow detailed information tag is expressed as: <folders>, for describing specific information of the workflow, such as workflow attribute, algorithm attribute, automatic function attribute, data synchronization attribute, data development attribute, and general component attribute. The attributes of the workflow specific information tag are fixed. Each attribute of the workflow specific information tag has a workFlowName, so when a new workflow is generated, the workFlowName of each attribute of the workflow specific information tag is replaced by the name of the new workflow.

[0108] wherein one workflow contains 6 parts, including: workflow name (Workflow name), algorithm (folder algm), user-defined function (folder User Defined), data synchronization (folder di), data development (folder Max Computer), and general component (folder General). All the available development components under the Dataworks data development page will appear in the folder (tag) corresponding to the data synchronization under the workflow specific information tag. Therefore, as long as a new workflow is created in Dataworks and a number of components are used, the information of these components will be stored under the workflow specific information tag. All SQL calculations based on the data warehouse exist in the folder corresponding to the data development under the workflow specific information tag.

[0109] The workflow node information tag is expressed as: <otherfiles>information of nodes in the workflow and the dependency relationship between the nodes. The workFlowName exists in the workflow node information tag, so when a new workflow is generated, the workFlowName in the workflow node information tag is replaced by the name of the new workflow.

[0110] The workflow creation script is executed to create the workflow basic information tag and the workflow specific information tag in the target XML file; the workflow creation script is executed to replace the workFlowName parameter of the workflow basic information tag with the target workflow name; and the parameters of the workflow specific information tag are replaced with the data processing service configuration data.

[0111] For S522, the workflow creation script is executed to create the workflow node information tag and the node specific information and node dependency relationship information in the workflow node information tag in the target XML file according to each target node name and the data processing service configuration data, and then the workFlowName of the node specific information and the node dependency relationship information in the workflow node information tag is replaced with the target workflow name.

[0112] In one embodiment, the step of creating the workflow node information tag and the node specific information and node dependency relationship information in the workflow node information tag in the target XML file according to the target workflow name, each target node name and the data processing service configuration data comprises:

[0113] S5221: The workflow node information tag is created in the target XML file by using a preset tag creation rule;

[0114] S5222: A node sub-tag is created in the workflow node information tag in the target XML file for each target node name;

[0115] S5223: A node specific information sub-tag, a node dependency relationship information sub-tag and a current node input / output information sub-tag are created in the node sub-tag in the target XML file according to each target node name and the workflow node information tag;

[0116] S5224: The attribute information of the node sub-tag is set in the target XML file according to each target node name and the data processing service configuration data;

[0117] S5225: In the target XML file, according to the data processing service configuration data, the node specific information sub-tag, the node dependency relationship information sub-tag and the current node input and output information sub-tag are set.

[0118] The embodiment realizes the creation of the workflow node information tag by using the preset tag creation rule, and then realizes the automatic creation and setting of the node specific information and the node dependency relationship information in the workflow node information tag, thereby avoiding manual creation, improving the creation efficiency and accuracy.

[0119] For S5221, the workflow creation script is executed, and in the target XML file, the workflow node information tag is created by using the preset tag creation rule, that is, the <otherfiles>.

[0120] For S5222, the workflow creation script is executed, and in the workflow node information tag in the target XML file, a preset sub-tag creation rule is used to create a node sub-tag for each target node name.

[0121] The workflow node information tag includes a plurality of node sub-tags, and each node sub-tag corresponds to a node. The node sub-tag of the workflow node information tag is expressed as otherFile.

[0122] For S5223, the node dependency relationship information sub-tag is expressed as <filerelations>, information for describing the dependency relationship between nodes. It can be understood that the node dependency relationship information sub-tag is mainly upstream dependency. Among them, ParentId (attribute) is the name of the upstream dependent task, and parentFileFolderPath (attribute) is the path of the upstream dependent task.

[0123] The node specific information sub-tag is expressed as <nodedef>, which is used to describe the specific information of the node, and the specific information of the node includes a node name, an output name of an upstream dependent node, and an output name of the node. Wherein, nodeName (attribute) and fileId (attribute) are the node name, input (attribute) is the output name of the upstream node, and output (attribute) is the output name of the node.

[0124] The node input and output information sub-tag is expressed as <nodeinputoutputs>, for describing input information and output information of the node.

[0125] The workflow creation script is executed, and the node specific information sub-tag, the node dependency relationship information sub-tag and the input and output information sub-tag of the node are created according to each target node name and the workflow node information tag in the node sub-tag in the target XML file according to a preset sub-tag creation rule.

[0126] For S5224, the attribute information of the node sub-tag includes a node name, a folder to which the node belongs and a workflow to which the node belongs. The folderPath attribute and the bizId attribute describe the workflow name, the fileName attribute and the nodeId attribute describe the node name, and the sourcePath attribute describes the path relationship between the workflow and the node.

[0127] The workflow creation script is executed, and the attribute information of the node sub-tag is set in the target XML file according to each target node name and the data processing service configuration data.

[0128] For S5225, the workflow creation script is executed, and the node specific information sub-tag, the node dependency relationship information sub-tag and the input and output information sub-tag of the node are set in the target XML file according to the data processing service configuration data, so that the creation of a single workflow is completed.

[0129] In one embodiment, the step of creating the fixed tags of the workflow description information in the target XML file according to the workflow XML generation rule includes:

[0130] S511: The compression type tag, the solution tag and the internal file tag are created in the target XML file according to the workflow XML generation rule.

[0131] In this embodiment, the compression type tag, the solution tag and the internal file tag are created according to the workflow XML generation rule, so that manual creation is avoided, and the efficiency and accuracy of creation are improved.

[0132] For S511, the workflow creation script is executed, and the compression type tag, the solution tag and the internal file tag are created in the target XML file according to the workflow XML generation rule.

[0133] Referring to Figure 2 The present application provides a workflow creation device of a data warehouse, which comprises:

[0134] The request acquisition module 100 is configured to acquire a workflow creation request, wherein the workflow creation request carries a business system identifier set.

[0135] The data acquisition module 200 is configured to acquire workflow table information data, data processing service configuration data, a workflow creation script and a target XML file in response to the workflow creation request.

[0136] The target business system identifier determination module 300 is configured to determine any business system identifier in the business system identifier set as a target business system identifier.

[0137] The workflow name and node name generation module 400 is configured to execute the workflow creation script, and generate a target workflow name and each target node name according to the target business system identifier, the workflow table information data and the data processing service configuration data.

[0138] The workflow description information creation module 500 is configured to execute the workflow creation script, and create workflow description information in the target XML file according to the target workflow name, each target node name and the data processing service configuration data.

[0139] The loop control module 600 is configured to repeatedly execute the step of determining any business system identifier in the business system identifier set as a target business system identifier until the acquisition of the business system identifiers in the business system identifier set is completed, and to take the target XML file as a workflow creation result.

[0140] The embodiment determines any business system identifier in the business system identifier set as a target business system identifier, executes the workflow creation script, generates a target workflow name and each target node name according to the target business system identifier, the workflow table information data and the data processing service configuration data, executes the workflow creation script, creates workflow description information in the target XML file according to the target workflow name, each target node name and the data processing service configuration data, repeatedly executes the step of determining any business system identifier in the business system identifier set as a target business system identifier until the acquisition of the business system identifiers in the business system identifier set is completed, and takes the target XML file as a workflow creation result, thereby realizing automatic batch workflow creation in an existing XML file, avoiding manual creation of nodes and configuration of node dependencies, improving the creation efficiency of the workflow and improving the accuracy of the workflow.

[0141] Reference Figure 3 In the embodiments of the present application, a computer device is also provided, which can be a server, and the internal structure of the computer device can be as shown in Figure 3 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data warehouse workflow creation method and other data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a data warehouse workflow creation method. The data warehouse workflow creation method includes: obtaining a workflow creation request, the workflow creation request carrying a set of business system identifiers; in response to the workflow creation request, obtaining workflow table information data, data processing service configuration data, a workflow creation script and a target XML file; taking any one of the set of business system identifiers as a target business system identifier; executing the workflow creation script to generate a target workflow name and each target node name according to the target business system identifier, the workflow table information data and the data processing service configuration data; executing the workflow creation script to create workflow description information in the target XML file according to the target workflow name, each target node name and the data processing service configuration data; repeating the step of taking any one of the set of business system identifiers as the target business system identifier until the business system identifiers in the set of business system identifiers are obtained, and taking the target XML file as a workflow creation result.

[0142] The embodiment executes the workflow creation script according to the target business system identifier, the workflow table information data and the data processing service configuration data to generate a target workflow name and each target node name, executes the workflow creation script, creates workflow description information in the target XML file according to the target workflow name, each target node name and the data processing service configuration data, repeatedly executes the step of taking any one of the business system identifiers in the business system identifier set as a target business system identifier until the acquisition of the business system identifiers in the business system identifier set is completed, takes the target XML file as a workflow creation result, thereby realizing automatic batch workflow creation in the existing XML file, avoiding manual creation of nodes and configuration of node dependencies, improving the workflow creation efficiency and improving the workflow accuracy.

[0143] An embodiment of the present application further provides a computer readable storage medium, which has a computer program stored thereon, and the computer program is executed by a processor to implement a workflow creation method of a data warehouse, including the steps that a workflow creation request is acquired, the workflow creation request carrying a business system identifier set; in response to the workflow creation request, workflow table information data, data processing service configuration data, a workflow creation script and a target XML file are acquired; any one of the business system identifiers in the business system identifier set is taken as a target business system identifier; the workflow creation script is executed to generate a target workflow name and each target node name according to the target business system identifier, the workflow table information data and the data processing service configuration data; the workflow creation script is executed to create workflow description information in the target XML file according to the target workflow name, each target node name and the data processing service configuration data; the step of taking any one of the business system identifiers in the business system identifier set as a target business system identifier is repeatedly executed until the acquisition of the business system identifiers in the business system identifier set is completed, and the target XML file is taken as a workflow creation result.

[0144] The workflow creation method of the data warehouse executed above, by taking any one of the business system identifiers in the set of business system identifiers as a target business system identifier, executes the workflow creation script, generates a target workflow name and each target node name according to the target business system identifier, the workflow table information data and the data processing service configuration data respectively, executes the workflow creation script, creates workflow description information in the target XML file according to the target workflow name, each target node name and the data processing service configuration data, repeats the step of taking any one of the business system identifiers in the set of business system identifiers as a target business system identifier until the acquisition of the business system identifiers in the set of business system identifiers is completed, takes the target XML file as a workflow creation result, thereby realizing automatic batch workflow creation in the existing XML file, avoiding manual creation of nodes and configuration of node dependencies, improving the efficiency of workflow creation and improving the accuracy of the workflow.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium provided by the present application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0146] It has to be noted that, as used herein, the terms "comprising", "including", "containing", etc. are to be interpreted as specifying the presence of the stated features, integers, steps or components but do not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof. The term "consisting essentially of to be interpreted as excluding any element other than the elements, integers, steps or components which are expressly recited. The term "consisting of to be interpreted as excluding the presence of anything other than the complement of the recited elements, integers, steps or components.

[0147] The preferred embodiments of the present application have been described above with the intent to enable those skilled in the art to make and use it. Various modifications to these embodiments will occur to those skilled in the art and are intended to be encompassed by the description. The structures and devices disclosed herein are to be considered in a descriptive sense only and not as limitations on the present application. Descriptions of examples using terms such as "comprises", "comprising", "containing" or "having" are not to be construed as indicating that the aspects of the application are limited to the specific features, integers, steps or components. Similarly, terms such as "comprises", "comprising", "containing" or "having" should not be interpreted as excluding the presence of elements, integers, steps or components that are not specifically mentioned.< / nodeinputoutputs> < / nodedef> < / filerelations> < / otherfiles> < / otherfiles> < / folders> < / bizinfos> < / innerfiles> < / solutions> < / ziptype>

Claims

1. A workflow creation method for a data warehouse, characterized in that, The method includes: Obtain a workflow creation request, wherein the workflow creation request carries a business system identifier set; In response to the workflow creation request, obtain workflow table information data, data processing service configuration data, workflow creation script, and target XML file; Use any one of the business system identifiers in the set of business system identifiers as the target business system identifier; Execute the workflow creation script to generate the target workflow name and the names of each target node based on the target business system identifier, the workflow table information data, and the data processing service configuration data, respectively. Execute the workflow creation script to create workflow description information in the target XML file based on the target workflow name, the names of each target node, and the data processing service configuration data. Repeat the step of taking any one of the business system identifiers in the business system identifier set as the target business system identifier until the acquisition of the business system identifiers in the business system identifier set is completed, and take the target XML file as the workflow creation result; The steps of generating the target workflow name and the names of each target node based on the target business system identifier, the workflow table information data, and the data processing service configuration data include: Using a preset workflow naming convention, a workflow name is generated based on the target business system identifier, the workflow table information data, and the data processing service configuration data to obtain the target workflow name; Obtain the workflow template, adopt the preset node naming standard, and generate node names based on the workflow template, the target business system identifier, the workflow table information data, and the data processing service configuration data to obtain the names of each target node. The target business system identifier corresponds to a target workflow name, and the target business system identifier corresponds to the target node name of a start node and the target node names of each process branch; The process branches correspond one-to-one with the data tables of the business systems corresponding to the target business system identifiers. Each process branch includes: a synchronous data buffer layer node, a synchronous data source layer node, and a synchronous data detail layer node.

2. The data warehouse workflow creation method according to claim 1, characterized in that, The step of creating workflow description information in the target XML file based on the target workflow name, the names of each target node, and the data processing service configuration data includes: In the target XML file, fixed tags for the workflow description information are created according to the workflow XML generation rules, wherein the fixed tags include: compression type tag, solution tag, and internal file tag; Below the fixed tag in the target XML file, basic information and process information of the workflow description information are created based on the target workflow name, the names of each target node, and the data processing service configuration data.

3. The data warehouse workflow creation method according to claim 2, characterized in that, The step of creating basic information and process information for the workflow description in the target XML file based on the target workflow name, the names of each target node, and the data processing service configuration data includes: In the target XML file, workflow basic information tags and workflow specific information tags are created according to the target workflow name and the data processing service configuration data, respectively; In the target XML file, workflow node information tags are created based on the target workflow name, the names of each target node, and the data processing service configuration data. The specific node information and node dependency information in the workflow node information tags are also created.

4. The data warehouse workflow creation method according to claim 3, characterized in that, The steps of creating workflow node information tags and creating specific node information and node dependency information in the workflow node information tags according to the target workflow name, the names of each target node, and the data processing service configuration data in the target XML file include: In the target XML file, the workflow node information tags are created using preset tag creation rules; In the workflow node information tag in the target XML file, node sub-tags are created for each target node name; In the node sub-tags of the target XML file, node specific information sub-tags, node dependency information sub-tags, and input / output information sub-tags of the current node are created based on the names of each target node and the workflow node information tags; In the target XML file, the attribute information of the node sub-tags is set according to the names of each target node and the data processing service configuration data; In the target XML file, based on the data processing service configuration data, the specific information sub-tags of the node, the dependency information sub-tags of the node, and the input / output information sub-tags of the current node are set.

5. The data warehouse workflow creation method according to claim 2, characterized in that, The step of creating fixed tags for the workflow description information in the target XML file according to the workflow XML generation rules includes: In the target XML file, compression type tags, solution tags, and internal file tags are created according to the workflow XML generation rules.

6. A workflow creation apparatus for a data warehouse, used to implement the method according to any one of claims 1-5, characterized in that, The device includes: The request acquisition module is used to acquire workflow creation requests, wherein the workflow creation requests carry a business system identifier set; The data acquisition module is used to respond to the workflow creation request and acquire workflow table information data, data processing service configuration data, workflow creation script and target XML file; The target business system identifier determination module is used to select any one of the business system identifiers in the set of business system identifiers as the target business system identifier; The workflow name and node name generation module is used to execute the workflow creation script and generate the target workflow name and each target node name according to the target business system identifier, the workflow table information data and the data processing service configuration data, respectively. The workflow description information creation module is used to execute the workflow creation script and create workflow description information in the target XML file based on the target workflow name, the names of each target node, and the data processing service configuration data. The loop control module is used to repeatedly execute the step of taking any one of the business system identifiers in the business system identifier set as the target business system identifier until the acquisition of the business system identifiers in the business system identifier set is completed, and the target XML file is taken as the workflow creation result.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images