A method and system for visualizing a data pipeline

Abstract

The present application relates to the field of data processing, and especially relates to a visual data pipeline generation method and system, the method comprising a forward process of converting an initial data flow graph into target language code and a reverse process of converting the target language code into the initial data flow graph, after obtaining the target language code depending on the forward or reverse process, storing the target language code into a preset database, and finally, in response to a trigger initiated by a user, converting the target language code stored in the database into a final data flow graph and performing a pipeline operation according to the final data flow graph. Compared with the prior art, the method of the present application solves the problem that data processing pipeline consistency is difficult to synchronize in the prior art.

Description

Technical Field

[0001] This invention relates to the field of data processing. More specifically, this invention relates to a method and system for generating a visual data pipeline. Background Technology

[0002] A data processing pipeline is a processing architecture that breaks down the data processing process into multiple ordered stages and uses automation mechanisms to enable the efficient and continuous flow of data from data sources to target systems. With the increasing demand for big data and real-time analytics, data processing pipeline systems are playing an increasingly important role in enterprise data architecture. In fields such as digital marketing, business intelligence, financial services, and the Internet of Things, it is often necessary to collect data from multiple data sources, perform real-time transformation and analysis, and output the results to different target systems. Therefore, how to efficiently and quickly generate domain-appropriate data processing pipelines is a current technological hotspot.

[0003] Existing pipeline systems primarily employ the following methods to generate data pipelines:

[0004] Firstly, data processing logic is defined by technical developers using programming languages ​​such as Java, Python, or Scala to form a data processing pipeline. This approach has a long development cycle, high technical threshold, high maintenance cost, and cannot be participated in by non-technical personnel. It requires repeated communication and confirmation between the product and technical sides. Overall, the generation efficiency of data pipelines is relatively low.

[0005] Secondly, while generating data pipelines through a visual interface overcomes the technical shortcomings of the above solutions, it suffers from a one-way generation problem. That is, after generating configuration code from the visualization, if the configuration code is refined, it is difficult to synchronize the visualization representation with the stored configuration. It cannot be restored to a visual data stream that can be edited / viewed by non-technical personnel, which leads to difficulties in the actual configuration of subsequent application scenarios and reduces the overall development efficiency.

[0006] Therefore, existing technologies mainly suffer from low efficiency in developing data processing pipelines and difficulty in synchronizing consistency. Summary of the Invention

[0007] To address the aforementioned technical problems of low development efficiency and difficulty in synchronizing consistency in data processing pipelines, this invention discloses a method and system for generating visual data pipelines.

[0008] In a first aspect, the visual data pipeline generation method disclosed in this invention includes:

[0009] The forward flow of converting the initial data flow diagram into target language code is as follows:

[0010] In response to user interaction actions, an initial data flow graph is generated; the initial data flow graph is located in the visualization interaction window;

[0011] Convert the initial data flow graph into target language code;

[0012] It also includes the reverse process of converting the target language code into the initial data flow graph, specifically:

[0013] Convert the target language code of the configuration changes into the user-change code;

[0014] In response to user feedback regarding code changes, update the initial data flow graph;

[0015] Also includes:

[0016] Store the target language code in a pre-defined database;

[0017] In response to a user-initiated trigger, the target language code stored in the database is converted into a final data flow graph, and a pipeline operation is executed based on the final data flow graph.

[0018] Preferably, the target language code stored in the database is converted into the final data flow graph, including:

[0019] Parse the configuration of the target language code and extract the operator arrays and the dependencies between them;

[0020] Based on the operator array and dependencies, construct the final data flow graph.

[0021] Preferably, converting the initial data flow graph into target language code includes:

[0022] Traverse all operator nodes of the initial data flow graph within the visualization interaction window;

[0023] Extract the ID, type, parameters, coordinates, and handle of each operator node;

[0024] Based on the operator node's ID, type, parameters, and coordinates, generate execution steps and metadata to be embedded;

[0025] Based on the handle, resolve the connection edges between two adjacent operator nodes and construct an upstream dependency array for each execution step;

[0026] Generate target language code based on the metadata to be embedded and the upstream dependency array.

[0027] Preferably, in response to a user's postback of a changed code, the initial data flow graph is updated, including:

[0028] Parse the user change code and extract the operation array;

[0029] Create node objects for the array being manipulated;

[0030] The initial data flow graph is reconstructed based on the upstream dependency array and the IDs of the node objects.

[0031] Preferably, user interaction actions include at least node creation, node dragging, and node connection operations.

[0032] Preferably, if the user interaction is a node drag operation, then in response to the triggering of the user interaction, an initial data flow graph is generated, including:

[0033] In response to the completion of a node drag operation, at least the ID and handle of the dragged child node should be resolved;

[0034] Generate an initial data flow graph based on the ID and handle of the dragged operator node.

[0035] Preferably, it also includes a connection processing and verification mechanism, specifically:

[0036] The system uses a pre-defined loop detection algorithm to detect whether loops exist in the initial or final data flow graph.

[0037] If so, delete the connections between the corresponding multiple operator nodes and issue the corresponding error message.

[0038] Preferably, if multiple operator nodes in the final data flow graph have no dependencies, a pipeline operation is performed based on the final data flow graph, including:

[0039] The final data flow graph is passed to a pre-defined distributed stream processing platform to split the final data flow graph into multiple jobs and distribute them to the corresponding task managers for distributed execution.

[0040] Preferably, if multiple operator nodes in the final data flow graph have dependencies, executing pipeline operations based on the final data flow graph further includes:

[0041] A preset topological sorting algorithm is used to perform topological sorting on the operator nodes with interdependent relationships in the final data flow graph to obtain the sorting result;

[0042] Based on the sorting results, execute the jobs corresponding to the operator nodes in topological order.

[0043] In a second aspect, the present invention discloses a visual data pipeline generation system for use with the visual data pipeline generation method described in the first aspect. The system of the present invention includes:

[0044] The visualization interface layer is used to generate an initial data flow diagram in response to user interaction actions; or to update the initial data flow diagram in response to user feedback of code changes; wherein, the initial data flow diagram is located in the visualization interaction window;

[0045] The transformation layer is used to convert the initial data flow diagram into target language code; or, to convert the target language code of configuration changes into user-modified code.

[0046] The service layer stores the target language code in a pre-defined database;

[0047] The execution layer, in response to user-initiated triggers, converts the target language code stored in the database into the final data flow graph and executes pipeline operations based on the final data flow graph.

[0048] The beneficial effects of this invention are as follows:

[0049] The method of this invention mainly utilizes visual image interaction to construct a data processing pipeline, effectively improving development efficiency. Based on this, the method incorporates a bidirectional conversion mechanism that supports both forward conversion from the initial data flow diagram to the target language code and reverse synchronous updates from the modified target language code to the target language code. This method effectively solves the problem of difficulty in synchronizing consistency in data processing pipelines in existing technologies. Attached Figure Description

[0050] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0051] Figure 1 This is a flowchart of the visual data pipeline generation method in Embodiment 1 of the present invention;

[0052] Figure 2 This is a schematic diagram of the structure of the visualized data pipeline generation system in Embodiment 2 of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0055] Example 1

[0056] like Figure 1As shown, this embodiment discloses a method for generating a visual data pipeline, including:

[0057] The forward flow of converting the initial data flow diagram into target language code is as follows:

[0058] S10: In response to user interaction actions, generate an initial data flow graph.

[0059] In this embodiment, the user interaction actions include at least node creation, node dragging, and node connection. All user interaction actions in this step are performed on the visual interaction window.

[0060] Specifically, users open the visual builder in a web browser, first dragging and dropping data source operator nodes (such as Kafka sources, database sources, and API sources) from the operator palette onto the canvas of the visual interaction window, and then configuring operator node parameters (such as connection information, query conditions, and field selection). Next, they add transformation operator nodes (such as mapping, filtering, or aggregation) and output operator nodes (such as database receivers and file receivers). Finally, they create directed edges between operator nodes by dragging connection handles, forming a complete initial data flow graph.

[0061] Furthermore, the aforementioned visual interactive window is implemented based on the Vue.js and VueFlow frameworks, providing users with more intuitive drag-and-drop operation guidance.

[0062] For the aforementioned operator palette: the operator node definitions are pre-serialized into JSON format and stored in a DataTransfer object for reading and recognition when the user drags operator nodes from the operator palette.

[0063] Regarding the canvas settings described above: VueFlow is used as the visual canvas to support features such as zooming, panning, multi-selection, and grid alignment. The canvas can maintain the reactive state of the node and edge arrays in real time.

[0064] When dragging and dropping a data source operator node from the operator palette onto the canvas of the visualization window, this implementation method mainly uses the onDrop event handler to perform the following steps: first, parse the JSON definition of the data source operator node from the corresponding DataTransfer object, then obtain the bounding rectangle of the canvas, then calculate the position of the mouse relative to the canvas, then use the project() function to convert the screen coordinates to flow coordinates, and finally adjust the position to center the operator node.

[0065] For each operator node on the canvas, this embodiment provides three types of node components: 1. Chart and label display component; 2. Edit and delete component; 3. Connection handle configuration component (Handle component).

[0066] It should be noted that each operator node on the canvas can be configured with multiple Handle components. The handles mentioned in this article include the left input handle (target-left) and the right output handle (source-right). An operator node contains a source node, a transformation node, and a receiver node. The source node only has a right output handle, the transformation node has both a left input handle and a right output handle, and the receiver node only has a left input handle.

[0067] Therefore, if the user interaction is a node drag operation, then step S100 includes:

[0068] S101: In response to the completion of a node drag operation, at least the ID and handle of the dragged operator node are resolved.

[0069] Here, ID refers to the unique identifier of the operator node, which is used to distinguish operator nodes.

[0070] S102: Generate the initial data flow graph based on the ID and handle of the dragged operator node.

[0071] Specifically, based on the operator node ID, the left input handle, and the right output handle, the operator nodes can be automatically connected to generate an initial data flow graph.

[0072] S11: Convert the initial data flow graph into target language code.

[0073] In this embodiment, the data exchange format for the target language code is JSON.

[0074] The specific steps of executing step S11 include:

[0075] S111: Traverse all operator nodes of the initial data flow graph within the visualization interaction window.

[0076] S112: Extract the ID, type, parameters, coordinates, and handle of each operator node.

[0077] The type is used to specify the class name of the operator node; the parameters of different types of operator nodes are different; the coordinates refer to the two-dimensional coordinates of the operator node on the canvas; the handle has been discussed clearly in the previous text and will not be repeated here.

[0078] S113: Generate execution steps and metadata to be embedded based on the ID, type, parameters and coordinates of the operator node.

[0079] Specifically, the number and sequence of execution steps can be determined based on the number and sequence number of IDs; and the IDs, their corresponding types, parameters, and coordinates are converted into metadata to be embedded. For example, its metadata structure is as follows:

[0080] {

[0081] "id": "kafka_source_1",

[0082] "class_name": "com.tecdo.dubhe.pipeline.processors.KafkaSource",

[0083] "upstream": [],

[0084] "params": { "topic": "events", "bootstrap.servers": "localhost:9092"},

[0085] "ui_metadata": {

[0086] "position_x": 150,

[0087] "position_y": 200

[0088] }

[0089] }

[0090] S114: Based on the handle, resolve the connection edges between two adjacent operator nodes and construct an array of upstream dependencies for each execution step.

[0091] In step S114, the parsed operator node cannot be the source node. The processing objects of step S114 are mainly the transformation node and the receiver node. For the transformation node and the receiver node, it is only necessary to determine the connection edge between them and the upstream operator node based on the left input handle.

[0092] S115: Generate target language code based on the metadata to be embedded and the upstream dependency array.

[0093] It should be noted that step S115 above is essentially the assembly process of the metadata to be embedded. Once the metadata to be embedded and the upstream dependency array of each operator node are determined, multiple operator nodes can be assembled at the code level.

[0094] For example, a complete target language code can be:

[0095] {

[0096] "name": "my-pipeline",

[0097] "operators": [

[0098] { "id": "source_1", "class_name": "...", "upstream": [], ...},

[0099] { "id": "map_1", "class_name": "...", "upstream": ["source_1"], ...},

[0100] { "id": "sink_1", "class_name": "...", "upstream": ["map_1"], ...} ]

[0102] }

[0103] Through the above steps S111-S115, the forward process of converting the initial data flow diagram into target language code can be realized.

[0104] It should be noted that the above steps primarily support the online creation, editing, and optimization of pipeline data processing flows by pipeline planners, thereby improving development efficiency. Based on this, the method in this embodiment also involves the reverse process of converting target language code into an initial data flow diagram, specifically:

[0105] S20: Convert the target language code of the configuration change into the user change code.

[0106] It should be explained that during the reverse conversion process, technicians directly use SSH or a terminal to connect to the server running PostgreSQL (database), and then modify the configuration file in the target language code to implement the configuration change. After the configuration change is completed, the original target language code will be identified as user-modified code and trigger the corresponding return update action.

[0107] It should be noted that after generating the initial data flow diagram, user editing on the canvas or direct editing of the target language code can all be done within the same interface. If the user edits on the canvas, the corresponding target language code is updated synchronously; if the user directly configures the target language code, the corresponding initial data flow diagram is updated synchronously in reverse. Each successful conversion is recorded in the relevant logs.

[0108] S21: In response to the user's code change feedback, update the initial data flow graph.

[0109] Specifically, step S21 above includes:

[0110] S211: Parse the user change code and extract the operation array.

[0111] In this embodiment, the above-mentioned operation array is mainly extracted by parsing the JSON code (user change code) configuration and operation logs.

[0112] S212: Create node objects for the array of operations.

[0113] For node reconstruction, this embodiment of the method is configured with an automated reconstruction script:

[0114] const node = {

[0115] id: op.id,

[0116] type: getOperatorType(op.class_name),

[0117] position: {

[0118] x: op.ui_metadata?.position_x || defaultX,

[0119] y: op.ui_metadata?.position_y || defaultY

[0120] },

[0121] data: {

[0122] label: op.id,

[0123] className: op.class_name,

[0124] params: op.params

[0125] }

[0126] }

[0127] Specifically, an automated reconstruction script is used to read data from the operation array, thereby enabling the creation of node objects.

[0128] S213: Reconstruct the initial data flow graph based on the upstream dependency array and the IDs of the node objects.

[0129] It should be noted that after a node object is created, its uniqueness is determined by its ID, and the connection between the node object and the original operator node is determined by the upstream dependency array, thereby facilitating the reconstruction of the initial data flow graph.

[0130] Through the aforementioned forward and reverse processes, the method in this embodiment achieves completely lossless round-trip conversion. That is, the same visualization configuration always generates the same JSON, and the modified JSON always reconstructs the same visualization representation. The execution and synchronization of the bidirectional process rely on the automatic operation of the script, effectively overcoming the problem of difficulty in synchronizing consistency in the prior art.

[0131] In addition to the above-described forward and reverse processes, the method in this embodiment also includes:

[0132] S30: Store the target language code in a preset database.

[0133] In this embodiment, the database mentioned above is PostgreSQL.

[0134] S40: In response to a user-initiated trigger, the target language code stored in the database is converted into a final data flow diagram and a pipeline operation is performed based on the final data flow diagram.

[0135] The step in step S40 above, which involves converting data to obtain the final data flow graph, is as follows:

[0136] S41: Parse the configuration of the target language code and extract the operator arrays and the dependencies between them.

[0137] It should be noted that the core execution engine of this embodiment uses FlinkPipelineBuilder, which is responsible for converting JSON configuration code into executable Flink jobs. After the target language code, i.e., the JSON configuration code, is input into the execution engine, it automatically extracts the operator arrays and the dependencies between multiple operator arrays from the JSON configuration.

[0138] S42: Construct the final data flow graph based on the operator array and dependencies.

[0139] It should be added that after obtaining the operator array and dependencies, the execution engine will automatically create / reuse environment functions and metadata caches applicable to the application environment to improve the applicability of the final data flow graph.

[0140] For example, the above environment function can be:

[0141] private static StreamExecutionEnvironment buildEnvironment() {

[0142] if (!environmentInitialized || sharedEnvironment == null) {

[0143] Configuration flinkConfig = new Configuration();

[0144] flinkConfig.set(RestOptions.PORT, 8082);

[0145] flinkConfig.set(RestOptions.ADDRESS, "0.0.0.0");

[0146] sharedEnvironment = StreamExecutionEnvironment

[0147] .createLocalEnvironmentWithWebUI(flinkConfig);

[0148] sharedEnvironment.setParallelism(1);

[0149] environmentInitialized = true;

[0150] }

[0151] return sharedEnvironment;

[0152] }

[0153] In step S40, the pipeline operation stage is performed according to the final data flow diagram. The method in this embodiment is mainly divided into two cases.

[0154] If multiple operator nodes in the final data flow graph have dependencies, then the above-mentioned execution pipeline stage includes:

[0155] S43: Use a preset topological sorting algorithm to perform topological sorting on the operator nodes with interdependent relationships in the final data flow graph, and obtain the sorting result. The topological sorting needs to ensure that:

[0156] 1. Ensure that all source operators (with an in-degree of 0) are executed first, i.e., source operators take precedence.

[0157] 2. Execution dependencies must satisfy the following condition: Each operator is executed only after all its upstream operators have completed.

[0158] 3. Determinism must be satisfied: the same pipeline configuration always produces the same execution order.

[0159] S44: Based on the sorting results, execute the jobs corresponding to the operator nodes in topological order.

[0160] The above-mentioned task is the Flink task mentioned earlier.

[0161] Conversely, if multiple operator nodes in the final data flow graph have no dependencies, then the above-mentioned execution pipeline stage includes:

[0162] S45: Pass the final data flow graph to the preset distributed stream processing platform to split the final data flow graph into multiple jobs and distribute them to the corresponding task managers for distributed execution.

[0163] In this embodiment, the distributed stream processing platform mentioned above can be Kafka. Distributed execution refers to parallel execution.

[0164] Through the above steps S10-S40, the method of this embodiment has the following advantages:

[0165] (1) The method in this embodiment provides a bidirectional lossless conversion mechanism that can achieve complete synchronization between the visual data flow graph and the JSON configuration.

[0166] (2) The method in this embodiment provides a technical solution based on strategy pattern and topology sorting algorithm to ensure the determinism of execution order.

[0167] (3) The method in this embodiment enables non-technical users to create complex data processing pipelines through drag-and-drop operations, overcoming the problem of high technical threshold.

[0168] Furthermore, between steps S10 and S40 above, the method of this embodiment also provides a connection processing and verification mechanism, specifically as follows:

[0169] S500: Detects whether there are loops in the initial or final data flow graph using a preset loop detection algorithm.

[0170] In this embodiment, loop detection is achieved using the DFS algorithm.

[0171] S600: If so, delete the connection between the corresponding multiple operator nodes and issue the corresponding error message.

[0172] S700: If not, receive the current update status of the initial or final data flow graph.

[0173] Through the above steps S500-S700, the method of this embodiment also introduces a loop detection mechanism, so that the generated initial data flow graph or final data flow graph is a directed acyclic graph, thereby avoiding the situation where multiple interdependent steps are executed simultaneously when performing topology sorting in S43, so that the generated initial data flow graph or final data flow graph conforms to industrial implementation standards.

[0174] It should be further noted that the above connection processing and verification mechanism also involves verification rules for user canvas interaction actions:

[0175] Rule 1: A source node cannot be defined as a receiver type;

[0176] Rule 2: The target node cannot be the source type.

[0177] If a user violates the above rules in defining the type, the verification will fail, an error message will be displayed, and the connection will be refused.

[0178] Example 2

[0179] like Figure 2 As shown, this embodiment discloses a visual data pipeline generation system for the visual data pipeline generation method described in Embodiment 1. The system includes:

[0180] The visualization interface layer is used to generate an initial data flow diagram in response to user interaction actions; or to update the initial data flow diagram in response to user feedback of code changes; wherein, the initial data flow diagram is located in the visualization interaction window;

[0181] The transformation layer is used to convert the initial data flow diagram into target language code; or, to convert the target language code of configuration changes into user-modified code.

[0182] The service layer stores the target language code in a pre-defined database;

[0183] The execution layer, in response to user-initiated triggers, converts the target language code stored in the database into the final data flow graph and executes pipeline operations based on the final data flow graph.

[0184] In the description of this specification, "multiple" means at least two, such as two, three or more, etc., unless otherwise expressly and specifically defined.

[0185] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A method for generating a visual data pipeline, characterized in that, include: The forward flow of converting the initial data flow diagram into target language code is as follows: In response to a user interaction, an initial data flow graph is generated; wherein the initial data flow graph is located in a visual interaction window; Convert the initial data flow graph into target language code; It also includes the reverse process of converting the target language code into the initial data flow graph, specifically: Convert the target language code of the configuration changes into the user-change code; In response to the feedback of the user change code, update the initial data flow graph; Also includes: The target language code is stored in a preset database; In response to a user-initiated trigger, the target language code stored in the database is converted into a final data flow graph, and a pipeline operation is performed based on the final data flow graph. Converting the initial data flow graph into target language code includes: Traverse all operator nodes of the initial data flow graph within the visualization interaction window; Extract the ID, type, parameters, coordinates, and handle of each operator node; Based on the ID, type, parameters, and coordinates of the operator node, the execution steps and metadata to be embedded are generated. Based on the handle, the connection edges between two adjacent operator nodes are parsed, and an upstream dependency array for each execution step is constructed; The target language code is generated based on the metadata to be embedded and the upstream dependency array; In response to a user's feedback regarding code changes, update the initial data flow graph, including: Parse the user change code and extract the operation array; Create node objects for the array of operations; The initial data flow graph is reconstructed based on the upstream dependency array and the ID of the node object.

2. The method for generating a visual data pipeline according to claim 1, characterized in that, Converting the target language code stored in the database into the final data flow graph includes: The configuration of the target language code is parsed, and the operator arrays and their dependencies are extracted. Based on the operator array and the dependencies, construct the final data flow graph.

3. The method for generating a visual data pipeline according to claim 1, characterized in that, The user interaction actions include at least node creation, node dragging, and node connection operations.

4. The method for generating a visualized data pipeline according to claim 3, characterized in that, If the user interaction action is a node drag operation, then in response to the triggering of the user interaction action, an initial data flow graph is generated, including: In response to the completion of a node drag operation, at least the ID and handle of the dragged child node should be resolved; Generate an initial data flow graph based on the ID and handle of the dragged operator node.

5. The method for generating a visual data pipeline according to claim 1, characterized in that, It also includes connection processing and verification mechanisms, specifically: The initial data flow graph or the final data flow graph is detected to have loops by a preset loop detection algorithm; If so, delete the connections between the corresponding multiple operator nodes and issue the corresponding error message.

6. The method for generating a visual data pipeline according to claim 1, characterized in that, If multiple operator nodes in the final data flow graph have no dependencies, a pipeline operation is executed based on the final data flow graph, including: The final data flow graph is transmitted to a preset distributed stream processing platform to split the final data flow graph into multiple jobs and distribute them to the corresponding task managers for distributed execution.

7. The method for generating a visual data pipeline according to claim 1, characterized in that, If multiple operator nodes in the final data flow graph have dependencies, the pipeline operation performed based on the final data flow graph also includes: A preset topological sorting algorithm is used to perform topological sorting on the operator nodes with interdependent relationships in the final data flow graph to obtain the sorting result; Based on the sorting results, the jobs corresponding to the operator nodes are executed in topological order.

8. A visual data pipeline generation system, characterized in that, The system for the visualization data pipeline generation method according to any one of claims 1-7 comprises: A visualization interface layer is used to generate an initial data flow graph in response to user interaction actions; or to update the initial data flow graph in response to user feedback of code changes; wherein the initial data flow graph is located in the visualization interaction window; A conversion layer is used to convert the initial data flow diagram into target language code; or, to convert the target language code of the configuration change into the user change code; The service layer stores the target language code in a preset database; The execution layer, in response to a user-initiated trigger, converts the target language code stored in the database into a final data flow graph and executes pipeline operations based on the final data flow graph.

Images