Data stream optimization system and method

Abstract

The present invention provides a method for optimizing a data stream, comprising: converting a plurality of Extract-Transform-Loads (ETLs) into a dataflow graph, the dataflow graph comprising a plurality of data processing courses, each data processing course comprising a plurality of data nodes, executing a first merge procedure comprising: repeatedly finding out at least two data processing courses having the same node, and combining the at least two data processing courses having the same node with the at least two data processing courses having the same node, the at least two data processing courses having the same node, the at least two data processing courses having the same node, the at least two data processing courses having the same node, and the at least two data processing courses having the same node; and merging the at least two data processing courses into a data family until any two of the data processing courses do not have the same node, executing a second merging program, including: calculating the column similarity of a node pair, the node pair being two adjacent data nodes in the data processing courses, the two data nodes are merged when the column similarity is greater than a threshold, and the dataflow graph is converted into an optimization script when the column similarity of the node pair is not greater than the threshold. According to the optimization method, the data flow diagram can be effectively and automatically generated.

Description

Technical Field

[0001] The present invention relates to data flow optimization, and in particular to a data flow optimization system and method. Background Art

[0002] In large organizations, the process of extracting, transforming, loading (ETL) data is complex because there are many heterogeneous data sources, different data structures, and multiple data representations need to be cleaned, integrated, and quality-assured for subsequent applications. Usually, the data flow is very lengthy before it becomes usable. The problems faced by existing technologies are as follows:

[0003] First, data flows can be better optimized: Building a data pipeline (a program, database, or other storage device used to access data flows) requires a lot of interviews and in-depth discussions, and the implementation time is prolonged due to constant revisions to real-world constraints. In addition, in large and complex organizations, data flows designed by humans have more redundancy and lower quality and cannot be optimized.

[0004] Second, it is difficult for humans to understand: Currently, Structured Query Language (SQL) or similar data processing methods are the mainstream, but it is difficult for humans to quickly understand how data pipelines are connected. Although the connection between data pipelines can be documented using the Data Governance (DG) process, it still needs to be created and updated manually, which is very time-consuming.

[0005] Third, it is difficult to track and assess the scope of impact: Without reading the script in detail, it is difficult to track where the error occurred, and it is also difficult to assess the number of data pipelines affected when a small number of defective data points are found downstream and need to be fixed.

[0006] Fourth, if an external solution is chosen, the existing data flow cannot be reused: Although some commercial software has a graphical user interface that can easily define or merge related data for analysis, the existing data pipeline cannot be reused, which increases labor costs. Summary of the invention

[0007] In view of this, the present invention proposes a data flow optimization system and method to solve the above problems.

[0008] According to an embodiment of the present invention, a data flow optimization method includes executing with a processor: using a conversion program to convert a plurality of extract-transform-load (ETL) scripts into a data flow graph, wherein the data flow graph includes a plurality of data processing processes (data lineages), each of the plurality of data processing processes includes a plurality of data nodes connected in a series, and each of the plurality of data nodes represents a data table; executing a first merging program to generate a plurality of data families (data family), wherein the first merging procedure comprises: repeatedly finding at least two data processing procedures with the same node in the multiple data processing procedures, and merging the at least two data processing procedures into one of the multiple data families based on the same node, until any two of the multiple data processing procedures do not have the same node; after the first merging procedure is executed, executing the second merging procedure for each of the multiple data processing procedures, wherein the second merging procedure comprises: calculating a column similarity of a node pair, wherein the node pair is two of the multiple data nodes that are adjacent in the series; and when the column similarity is greater than a threshold value, merging the two data nodes into one node; and when the similarity of the node pair is not greater than the threshold value, converting the data flow graph into an optimization script.

[0009] According to an embodiment of the present invention, a data flow optimization system includes a storage device and a computing device. The storage device is used to store a plurality of extract-transform-load scripts and a plurality of instructions. The computing device is electrically connected to the storage circuit. The computing circuit is used to execute the plurality of instructions to perform a plurality of operations, wherein the plurality of operations include: using a conversion program to convert the plurality of extract-transform-load (ETL) scripts into a data flow graph, wherein the data flow graph includes a plurality of data processing processes (data lineage), each of the multiple data processing processes includes multiple data nodes connected in a series, and each of the multiple data nodes represents a data table; a first merging program is executed to generate multiple data families, wherein the first merging program includes: repeatedly finding at least two data processing processes with the same node in the multiple data processing processes, and merging at least two data processing processes into one of the multiple data families based on the same node, until any two of the multiple data processing processes do not have the same node; after the first merging program is executed, a second merging program is executed for each of the multiple data processing processes, wherein the second merging program includes: calculating a column similarity of a node pair, wherein the node pair is two of the multiple data nodes that are adjacent in the series; and when the column similarity is greater than a threshold, merging the two data nodes into one node; and when the column similarity of the node pair is not greater than the threshold, converting the data flow graph into an optimization script.

[0010] In summary, the data flow optimization system and method proposed by the present invention effectively reduces the number of data tables, thereby reducing the hardware cost of storing these data tables, and improving the computing device after the data tables are loaded. The present invention can automatically generate a data flow graph efficiently and has the following effects:

[0011] First, automation and optimization: The system automatically generates designed data flows, allowing users to focus on defining their needs for business operations or data applications. The data flow diagram optimizes and integrates complex data input and output ranges. In addition, process automation means that it is easier and faster to modify processes, regenerate data, and test on the data pipeline.

[0012] Second, visualization and documentation: Visualization of data processing history helps human understanding, especially when dealing with missing or misaligned data. The invention also ensures that the generated documents meet the requirements of data governance. On the other hand, from the perspective of data analysis, since the data is digitized, users can use a graphical user interface to design or fine-tune the required data flow.

[0013] Third, support for impact analysis: The generated data processing history is based on column-wise connections. Therefore, when debugging or updating the data flow, the relevant data pipeline can be easily highlighted for human understanding, which also means that revisions can be performed automatically under human supervision.

[0014] Fourth, reusability: These improved data flows are compatible with legacy systems because existing data flows can be included in system optimization and integration.

[0015] The above description of the content of the present invention and the following description of the implementation modes are used to demonstrate and explain the spirit and principle of the present invention, and to provide a further explanation of the scope of the patent application of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram of a data flow optimization system according to an embodiment of the present invention;

[0017] Figure 2 is a flow chart of a data stream optimization method according to an embodiment of the present invention;

[0018] Figure 3 It is an example of a data flow diagram;

[0019] Figure 4 It is an example of a data family;

[0020] Figure 5 is a flow chart of a first merging procedure according to an embodiment of the present invention;

[0021] Figure 6 is a flow chart of a second merging procedure according to an embodiment of the present invention;

[0022] Figure 7 is a flow chart of calculating column similarity according to an embodiment of the present invention;

[0023] Figure 8 is a flow chart of a data stream optimization method according to another embodiment of the present invention; and

[0024] Fig. 9 is a flowchart of a third merging procedure according to an embodiment of the present invention.

[0025] Component number description

[0026] 10 System

[0027] 1 Storage device

[0028] 3 Computing device

[0029] S1-S5, S21-S23, S31-S34, S311-S313, steps

[0030] P1, P11-P18

[0031] 121, 169, 31, 84 data nodes DETAILED DESCRIPTION

[0032] The detailed features and characteristics of the present invention are described in detail in the following embodiments, and the content is sufficient to enable any person skilled in the relevant art to understand the technical content of the present invention and implement it accordingly, and according to the content disclosed in this specification, the scope of the patent application and the drawings, any person skilled in the relevant art can easily understand the relevant concepts and characteristics of the present invention. The following examples are used to further illustrate the viewpoints of the present invention in detail, but are not intended to limit the scope of the present invention in any way.

[0033] Figure 1 FIG. 1 is a block diagram of a data flow optimization system according to an embodiment of the present invention. Figure 1 As shown, the system 10 includes a storage device 1 and a computing device 3. The storage device 1 is used to store an extract-transform-load (ETL) script and a plurality of instructions. The computing device 3 is electrically connected to the storage circuit, and the computing circuit is used to execute the plurality of instructions to perform a plurality of operations. The corresponding processes of these operations are shown in FIG. Figure 2 .

[0034] In one embodiment, the storage device 1 is, for example, a volatile memory and / or a non-volatile memory. The non-volatile memory includes a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable and programmable ROM (EEPROM), a flash memory, a phase-change random access memory (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM) and / or a ferroelectric RAM (FRAM). The volatile memory may include a dynamic RAM (DRAM), a static RAM (SRAM) and / or a synchronous DRAM (SDRAM). In another embodiment, the storage device 1 is, for example, at least one of a hard disk drive (HDD), a solid-state drive (SSD), a compact flash (CF) card, a secure digital (SD) card, a micro SD card, a mini SD card, an extreme digital (xD) card, and a memory stick.

[0035] In one embodiment, the computing device 3 may be implemented using one or more of the following examples: a personal computer, a network server, a microcontroller (MCU), an application processor (AP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system-on-a-chip (SOC), a deep learning accelerator, or any electronic device with similar functions. The present invention does not limit the hardware type of the computing device 3.

[0036] Figure 2 FIG. 1 is a flow chart of a method for optimizing a data stream according to an embodiment of the present invention. Figure 2As shown, the method includes steps S1 to S5.

[0037] In step S1 , the computing device 3 executes a conversion program to convert an extract-transform-load (ETL) script into a data flow graph.

[0038] In one embodiment, the ETL script is a Structured Query Language script (SQL script), and the conversion program is SQLLineage or Python SQLparser. However, the present invention is not limited to the above examples. The multiple ETL scripts are used to operate multiple source data tables in the database and convert them into target data tables.

[0039] Figure 3 is an example of a data flow graph. Figure 3 As shown, the data flow graph includes multiple data processing processes (datalineage). Each data processing process includes multiple data nodes connected in a series. Each data node represents a data table, as shown in FIG. Figure 3 The value in the circle represents the data table ID, and the link between two data nodes represents the relationship between the data tables. The relationship is, for example, taking two of the three fields included in table 121 as table 69. Figure 3 The nodes 121 and 53 in the table can be regarded as source data tables, and the nodes 84 and 173 can be regarded as target data tables.

[0040] In the initial stage of the requirements analysis of the Data Governance (DG) process, the input and output schemas are predefined. However, as the data pipeline grows, multiple similar data domains (columns) or data tables, or repeated data flow subgraphs may be generated. The goal of the present invention is to integrate them. Existing data nodes are regrouped using an unsupervised clustering method based on the following features: content similarity (based on data domains), traversal similarity (based on data tables), granularity (based on data domains), connections to output nodes, and field attributes (such as numeric values ​​or strings).

[0041] In step S2 , the computing device 3 executes a first merging procedure to reduce the number of data nodes (data tables) by aggregating multiple data processing processes. The aggregation result is called a data family. Figure 4 The example presents a data family based on Figure 3The result of the first merge procedure is as follows:

[0042] Specifically, the first merger procedure is as follows: Figure 5 The flowchart is shown in FIG.

[0043] Step S21, finding at least two data processing processes with the same node in a plurality of data processing processes. Figure 3 Node 169 and node 31 are shown.

[0044] Step S22: merging the at least two data processing processes into one data cluster according to the same node.

[0045] Step S23, determine whether there are any identical nodes in the remaining data processing histories. If the determination is yes, return to step S21, if the determination is no, proceed to step S3. In other words, the first merging procedure is repeatedly executed until no two of all data processing histories have identical nodes. After the first merging procedure in step S2 is completed, proceed to step S3.

[0046] In step S3, the computing device 3 executes a second merging procedure for each data processing process to reduce the number of similar data nodes in a single data processing process. Figure 6 The flowchart is shown in FIG.

[0047] In step S31, the column similarity of a pair of nodes is calculated. A pair of nodes is two adjacent data nodes in a series. Figure 3 For example, node 121 and node 169 form a node pair, and node 169 and node 31 form another node pair.

[0048] Figure 7 is a flow chart of calculating column similarity according to an embodiment of the present invention, including steps S311 to S313.

[0049] In step S311 , the computing device 3 obtains the intersection between a plurality of first columns of a first data table and a plurality of second columns of a second data table, wherein the first data table corresponds to one node in the node pair and the second data table corresponds to the other node in the node pair.

[0050] In step S312 , the computing device 3 obtains a union between a plurality of first columns of the first data table and a plurality of second columns of the second data table.

[0051] In step S313 , the computing device 3 divides the intersection by the union to generate the column similarity.

[0052] The first data table and the second data table correspond to two adjacent data nodes in the series. Figure 3 For example, assume that node 121 is a first data table, including three first columns: {C1, C2, C3}. Node 169 is a second data table, including three second columns: {C2, C3, C4}.

[0053] Therefore, the intersection obtained in step S311 is {C2, C3}, and the union obtained in step S322 is {C1, C2, C3, C4}. The column similarity calculated in step S313 is 2 / 4=0.5.

[0054] In step S32, the computing device 3 determines whether the column similarity is greater than a first threshold. According to the above calculation method, the larger the similarity value is, the higher the similarity between the two data tables is. In one embodiment, the first threshold is between 0 and 1, and the present invention does not specifically limit its value setting.

[0055] In step S33, since the column similarity is greater than the first threshold, the computing device 3 merges the two data nodes of the node pair into one node. In one embodiment, the merging method is to retain one of the two data nodes and delete the other one.

[0056] In step S34, the computing device 3 determines whether there are any node pairs whose column similarity has not been calculated. If the determination is yes, the next node pair is selected and the process returns to step S31. If the determination is no, the process continues to step S4.

[0057] In step S4, the computing device 3 determines whether the column similarities of all node pairs are not greater than the first threshold value. If it is determined to be yes, the process proceeds to step S5, and if it is determined to be no, the process returns to step S3.

[0058] In step S5 , the computing device 3 executes a conversion program to convert the optimized data flow graph back into an ETL script.

[0059] exist Figure 2 In the embodiment of the data stream optimization method shown, repeated data tables and similar data tables in the data stream can be effectively reduced through the first merging procedure and the second merging procedure.

[0060] Figure 8 is a flow chart of a data flow optimization method according to another embodiment of the present invention. Figure 2 An embodiment of Figure 8 The embodiment adds step P1 after step S3. In step P1, the computing device 3 performs a third merging procedure on each of the data clusters to reduce the number of similar data domains.

[0061] The data family used to execute the third merging procedure each time is called a candidate data family. A candidate data family includes multiple candidate data nodes. Each candidate data node includes multiple text columns and multiple value columns. The third merging procedure first starts with the integration of the text columns. The example in Table 1 below presents multiple text columns contained in each of the multiple candidate data nodes. For example, candidate data node 1 includes only one text column: BU. Candidate data node 2 includes four text columns: MANUFACTURE, CLASS_CODE_PROC_COMMODITY, CLASSCODE, and PCA. Because there is no text column similar to BU, the first cell of candidate data node 2 is blank.

[0062] Table 1. Example of text column for candidate data node.

[0063]

[0064]

[0065] Fig. 9 is a flowchart of a third merging procedure according to an embodiment of the present invention, including steps P11 to P18.

[0066] In step P11, the computing device 3 determines the dimension of the text column of the candidate data node. In the process of data governance, different people use data tables in different ways, resulting in the same text column being modified into different forms. Therefore, in one embodiment, as shown in Table 1, all text columns of all candidate data nodes are listed first, and then the dimension corresponding to each text column can be manually determined in real time. In another embodiment, the dimension corresponding to each text column can be automatically determined based on a pre-defined dimension setting table. Table 2 below is an example of a dimension setting table.

[0067] Table 2. Example of dimension setting table.

[0068]

[0069]

[0070] For example, according to Table 2, the text column BU of candidate data node 1 and the text column BU_CODE of candidate data node 3 correspond to the dimension "organization"; the text column MANUFACTURE of candidate data node 2 and the text column VENDER of candidate data node 3 correspond to the dimension "supplier", and so on. Therefore, the computing device 3 can find the dimension corresponding to the text column in the dimension setting table according to the content of the text column through a string comparison algorithm.

[0071] In step P12, the computing device 3 generates a plurality of groups according to the dimension, each group includes candidate nodes corresponding to the text fields of the same dimension. Based on Table 1 and Table 2, Table 3 below is an example of the grouping result.

[0072] Table 3, example of grouping results.

[0073] Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Group 1 organize organize organize organize Group 2 Manufacturer Manufacturer Manufacturer Manufacturer Group 3 Category Code Category Code Category Code Group 4 Material Material Material

[0074] As shown in Table 3, candidate data nodes 1, 3, 5, and 6 in group 1 all have the same dimension, "Organization". Candidate data nodes 2, 3, 4, and 5 in group 2 all have the same dimension, "Manufacturer". It is worth noting that in Table 1, candidate data node 2 originally had two different text columns, CLASS_CODE_PROC_COMMODITY and CLASSCODE. However, according to the dimension setting table in Table 2, these two text columns belong to the same dimension, "Category Code". Therefore, these two text columns can be merged into one later.

[0075] In step P13, the computing device 3 deletes the group whose loss ratio is not less than the second threshold. The loss ratio is the ratio of the number of elements in the group divided by the number of candidate data nodes. Taking Table 3 as an example, the loss ratio of group 1 is 4 / 6≒0.66. The loss ratio of group 3 is 3 / 6=0.5. Therefore, if the second threshold is set to 0.5, groups 3 and 4 are deleted.

[0076] In step P14, the computing device 3 selects candidate data nodes based on the intersection of the dimensions in the remaining groups, and adds the mergeable text fields in the candidate data nodes to the integrated data table. Continuing from the previous example, after step P13, in the remaining groups 1 and 2, the intersection of the dimensions is "organization" and "manufacturer", and the candidate data nodes corresponding to these two dimensions are nodes 3 and nodes 5. Therefore, the computing device 3 selects mergeable text fields in nodes 3 and 5. For example: two text fields such as BU_CODE of node 3 and BU of node 5 can be merged. Four text fields such as VENDOR, MANUFACTURE_AVLID of node 3, VENDOR, MANUFACTURE_AVLID of node 5 can be merged. In one embodiment, the merge refers to retaining one of them. For example, BU of node 5 and VENDER of node 3 are retained in the integrated data table. Table 4 below is an example of an integrated data table.

[0077] Table 4. Example of integrated data table.

[0078]

[0079] In step P15, the computing device 3 calculates the correlation between the two value columns of the selected candidate data node. Assuming that the candidate data node 3 includes three value columns M31, M32 and M33, and the candidate data node 5 includes two value columns M51 and M52, the computing device needs to calculate six correlations, including M31 and M51, M31 and M52, M32 and M51, M32 and M52, M33 and M51, and M33 and M52.

[0080] In step P16, if the calculated correlation is greater than the third threshold value (e.g., 0.9), then proceed to step P17, merge the two value columns into one value column and add it to the integrated data table, as shown in "Integrated Value Column" of Table 4. On the other hand, if the correlation between the two value columns is not greater than the third threshold value, then proceed to step P18, add the two value columns to the integrated data table, as shown in "Node Three Value Column" and "Node Five Value Column" of Table 4.

[0081] Depend on Fig. 9 It can be seen that the third merging procedure first determines which data tables have the same dimension in the text columns and can be integrated, and then finds the highly correlated value columns in these data tables for integration, and finally generates a large integrated data table. By executing the third merging procedure for each data family, the number of data domains can be effectively reduced.

[0082] In summary, the data flow optimization system and method proposed by the present invention effectively reduces the number of data tables, thereby reducing the hardware cost of storing these data tables, and improving the instruction cycle after the data tables are loaded. The present invention can automatically generate a data flow graph efficiently and has the following effects:

[0083] First, automation and optimization: The system automatically generates designed data flows, allowing users to focus on defining their needs for business operations or data applications. The data flow diagram optimizes and integrates complex data input and output ranges. In addition, process automation means that it is easier and faster to modify processes, regenerate data, and test on the data pipeline.

[0084] Second, visualization and documentation: Visualization of data processing history helps human understanding, especially when dealing with missing or misaligned data. The invention also ensures that the generated documents meet the requirements of data governance. On the other hand, from the perspective of data analysis, since the data is digitized, users can use a graphical user interface to design or fine-tune the required data flow.

[0085] Third, support for impact analysis: The generated data processing history is based on column-wise connections. Therefore, when debugging or updating the data flow, the relevant data pipeline can be easily highlighted for human understanding, which also means that revisions can be performed automatically under human supervision.

[0086] Fourth, reusability: These improved data flows are compatible with legacy systems because existing data flows can be included in system optimization and integration.

[0087] Although the present invention is disclosed as above with the aforementioned embodiments, it is not intended to limit the present invention. Without departing from the spirit and scope of the present invention, all changes and modifications are within the scope of patent protection of the present invention. Please refer to the attached patent application for the scope of protection defined by the present invention.

Claims

1. A data flow optimization method, characterized in that: The method comprises executing, with a processor: Converting the plurality of extract-transform-load scripts into a data flow graph using a conversion program, wherein the data flow graph includes a plurality of data processing procedures, each of the plurality of data processing procedures includes a plurality of data nodes connected in a series, and each of the plurality of data nodes represents a data table; Executing a first merging procedure to generate a plurality of data families, wherein the first merging procedure comprises: repeatedly finding at least two data processing procedures having the same node in the plurality of data processing procedures, and merging the at least two data processing procedures into one of the plurality of data families according to the same node, until any two of the plurality of data processing procedures do not have the same node; After the first merging procedure is executed, a second merging procedure is executed for each of the plurality of data processing procedures, wherein the second merging procedure comprises: calculating a column similarity of a node pair, wherein the node pair is two data nodes in the plurality of data nodes that are adjacent in the series; and when the column similarity is greater than a first threshold, merging the two data nodes into one node; and When the column similarity of the node pair is not greater than the first threshold, the data flow graph is converted into an optimization script.

2. The data stream optimization method according to claim 1, characterized in that: The method further includes: after the second merging procedure is completed, executing a third merging procedure on a candidate data family, wherein the candidate data family is each of the plurality of data families, the candidate data family includes a plurality of candidate data nodes, each of the plurality of candidate data nodes includes a plurality of text fields and a plurality of value fields, and the third merging procedure includes: Determining, according to a dimension setting table, a plurality of dimensions corresponding to each of the plurality of text columns; generating a plurality of groups according to the plurality of dimensions, wherein each of the plurality of groups includes at least one node, the at least one node is a part of the plurality of candidate data nodes, and one of the plurality of text columns corresponding to the at least one node has the same dimension; Deleting the plurality of groups whose loss ratio is not less than a second threshold, wherein the loss ratio is a ratio of the number of elements in each of the plurality of groups to the number of the plurality of candidate data nodes; In the remaining plurality of groups, a portion of the plurality of candidate data nodes is selected according to an intersection of the plurality of dimensions, and the plurality of text fields corresponding to the portion are added to the integrated data table; calculating correlations between the plurality of value columns of the portion; When the correlation is greater than a third threshold, merging two of the plurality of value columns and adding them into an integrated data table; and When the correlation is not greater than the third threshold, two of the plurality of value columns are added to the integrated data table.

3. The data stream optimization method according to claim 1, characterized in that: Calculating the column similarity of the node pair includes: Obtaining intersections between a plurality of first columns of a first data table and a plurality of second columns of a second data table, wherein the first data table and the second data table correspond to two of the plurality of adjacent data nodes in the series respectively; obtaining a union between the plurality of first columns and the plurality of second columns; and The intersection is divided by the union to produce the column similarity.

4. The data stream optimization method according to claim 1, characterized in that: The plurality of extract-transform-load scripts are structured query language scripts.

5. The data stream optimization method according to claim 1, characterized in that: The conversion program is SQLLineage or python SQLparser.

6. A data flow optimization system, characterized in that: include: a storage device for storing a plurality of extraction-conversion-load scripts and a plurality of instructions; as well as a computing device electrically connected to the storage circuit, the computing circuit being used to execute the plurality of instructions to perform a plurality of operations, the plurality of operations comprising: Converting the plurality of extract-transform-load scripts into a data flow graph using a conversion program, wherein the data flow graph includes a plurality of data processing procedures, each of the plurality of data processing procedures includes a plurality of data nodes connected in a series, and each of the plurality of data nodes represents a data table; Executing a first merging procedure to generate a plurality of data families, wherein the first merging procedure comprises: repeatedly finding at least two data processing procedures having the same node in the plurality of data processing procedures, and merging the at least two data processing procedures into one of the plurality of data families according to the same node, until any two of the plurality of data processing procedures do not have the same node; After the first merging procedure is executed, a second merging procedure is executed for each of the plurality of data processing procedures, wherein the second merging procedure comprises: calculating a column similarity of a node pair, wherein the node pair is two data nodes in the plurality of data nodes that are adjacent in the series; and when the column similarity is greater than a first threshold, merging the two data nodes into one node; and When the column similarity of the node pair is not greater than the first threshold, the data flow graph is converted into an optimization script.

7. The data stream optimization system according to claim 6, characterized in that: The plurality of operations further include: after the second merging procedure is executed, executing a third merging procedure on a candidate data family, wherein the candidate data family is each of the plurality of data families, the candidate data family includes a plurality of candidate data nodes, each of the plurality of candidate data nodes includes a plurality of text fields and a plurality of value fields, and the third merging procedure includes: Determining, according to a dimension setting table, a plurality of dimensions corresponding to each of the plurality of text columns; generating a plurality of groups according to the plurality of dimensions, wherein each of the plurality of groups includes at least one node, the at least one node is a part of the plurality of candidate data nodes, and one of the plurality of text columns corresponding to the at least one node has the same dimension; Deleting the plurality of groups whose loss ratio is not less than a second threshold, wherein the loss ratio is a ratio of the number of elements in each of the plurality of groups to the number of the plurality of candidate data nodes; In the remaining plurality of groups, a portion of the plurality of candidate data nodes is selected according to an intersection of the plurality of dimensions, and the plurality of text fields corresponding to the portion are added to the integrated data table; calculating correlations between the plurality of value columns of the portion; When the correlation is greater than a third threshold, merging two of the plurality of value columns and adding them into an integrated data table; and When the correlation is not greater than the third threshold, two of the plurality of value columns are added to the integrated data table.

8. The data stream optimization system according to claim 6, characterized in that: In the plurality of operations, calculating the column similarity of the node pair includes: Obtaining intersections between a plurality of first columns of a first data table and a plurality of second columns of a second data table, wherein the first data table and the second data table correspond to two of the plurality of adjacent data nodes in the series respectively; obtaining a union between the plurality of first columns and the plurality of second columns; and The intersection is divided by the union to produce the column similarity.

9. The data stream optimization system according to claim 6, characterized in that: The plurality of extract-transform-load scripts are structured query language scripts.

10. The data stream optimization system according to claim 6, characterized in that: The conversion program is SQLLineage or python SQLparser.

Images