Data analysis method and device based on directed acyclic graph and storage medium

Abstract

The invention discloses a data analysis method and device based on a directed acyclic graph and a storage medium, and relates to the technical field of data processing, the method comprises the steps that SQL statements in a data processing operation script are converted into minimum processing units SDP, then the SQL-level directed acyclic graph is constructed according to the grammar operation relation between the minimum processing units SDP, and the data analysis efficiency is improved. The method comprises the steps that firstly, SQL-level directed acyclic graphs are constructed, then table dependency relationships between the SQL-level directed acyclic graphs are analyzed, a job-level directed acyclic graph is constructed on the basis of the table dependency relationships, finally, a link-level directed acyclic graph is constructed according to the job dependency relationships corresponding to the job-level directed acyclic graph, and data operation model operation is executed on the link-level directed acyclic graph. And generating and outputting the optimized link-level directed acyclic graph. A complex data processing flow is decomposed into minimum processing units (SDP), a directed acyclic graph is constructed, and data operation model operation is performed on the directed acyclic graph, so that the accuracy of complex data link analysis optimization is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data analysis method, device, and storage medium based on a directed acyclic graph. Background Art

[0002] With the surge in data volume and the increasing complexity of processing links, efficient analysis and optimization of data model logic have become the key to improving data processing efficiency.

[0003] Currently, traditional solutions typically extract table-level input-output relationships based on database logs, job scripts, or data catalog tools to generate a cross-job table lineage graph. These solutions then analyze the SELECT field list and JOIN conditions to establish direct reference relationships between fields. However, when faced with massive and complex links, these solutions can only display data dependencies and are unable to analyze the specific processing logic between fields. This results in low accuracy in analyzing and optimizing complex data links.

[0004] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The present application provides a data analysis method, device and storage medium based on a directed acyclic graph, aiming to solve the problem of low accuracy of traditional solutions in analyzing and optimizing complex data links.

[0006] To achieve the above objectives, the present application provides a data analysis method based on a directed acyclic graph, which comprises the following steps:

[0007] Convert each SQL statement in the data processing script into the minimum processing unit SDP;

[0008] Constructing a SQL-level directed acyclic graph according to the grammatical operation relationship between the minimum processing units SDP;

[0009] Parsing table dependencies between each of the SQL-level directed acyclic graphs, and constructing a job-level directed acyclic graph based on the table dependencies;

[0010] Constructing a link-level directed acyclic graph based on the job dependencies corresponding to the job-level directed acyclic graph;

[0011] A data operation model operation is performed on the link-level directed acyclic graph to generate and output an optimized link-level directed acyclic graph.

[0012] In one embodiment, the step of constructing an SQL-level directed acyclic graph based on the grammatical operation relationship between the minimum processing units SDP includes:

[0013] When parsing the minimum processing unit SDP, if the minimum processing unit SDP contains subqueries, the subqueries are independently decomposed into nested basic data operation sets, and mapped layer by layer to the corresponding minimum processing units;

[0014] The output matrix of the minimum processing unit SDP corresponding to the sub-query is used as the input matrix of the main query;

[0015] According to the basic data operations of the main query, a SQL-level directed acyclic graph including sub-query data processing units and main query data processing units is constructed, wherein the sub-query data processing units are embedded in the hierarchical structure of the main query as independent nodes.

[0016] In one embodiment, the step of parsing the table dependency relationship between each of the SQL-level directed acyclic graphs and constructing the job-level directed acyclic graph based on the table dependency relationship further includes:

[0017] parsing the table dependencies of each of the SQL-level directed acyclic graphs;

[0018] In the statement pairs with the table dependency relationship, the output data matrix of the minimum data processing unit SDP corresponding to the upstream statement is used as the input data matrix of the unit corresponding to the downstream statement;

[0019] Determining an input-output relationship between the output data matrix and the input matrix, and connecting at least one of the SQL-level directed acyclic graphs into a job-level directed acyclic graph according to the input-output relationship;

[0020] If there are independent data processing statements without dependencies in the data processing link, the corresponding target SQL-level directed acyclic graph is retained and used as a subgraph of the job-level directed acyclic graph.

[0021] In one embodiment, the step of parsing the table dependencies of each of the SQL-level directed acyclic graphs includes:

[0022] Traversing a list of minimum processing units in the SQL-level directed acyclic graph;

[0023] Constructing a computational model of the SQL-level directed acyclic graph according to the minimum processing unit list;

[0024] Obtain a source table in the SQL-level directed acyclic graph, and query a target computing model corresponding to the source table;

[0025] A parent-child relationship of model nodes is constructed based on the target operation model, and the parent-child relationship is determined as the table dependency relationship.

[0026] In one embodiment, the step of performing a data operation model operation on the link-level directed acyclic graph to generate and output an optimized link-level directed acyclic graph includes:

[0027] Traversing the output field of each minimum processing unit SDP in the link-level directed acyclic graph, and marking the output field as a redundant field if the output field is not referenced by a subsequent minimum processing unit SDP and is not used as a final output target field;

[0028] If all output fields of the target minimum processing unit are marked as redundant fields, the target minimum processing unit is marked as a redundant minimum processing unit;

[0029] Based on the dependency path of the output fields in the link-level directed acyclic graph, redundant fields and / or redundant minimum processing units that do not participate in the final output are removed to complete the data calculation model operation on the directed acyclic graph.

[0030] In one embodiment, after the step of marking the target minimum processing unit as a redundant minimum processing unit if all output fields of the target minimum processing unit are marked as redundant fields, the method further includes:

[0031] If the output data matrices of two minimum processing units SDP are the same data object, then the two minimum processing units SDP are merged into a single node and their input-output dependency is updated;

[0032] When removing redundant minimum processing units, if the output data matrix of the redundant minimum processing unit is relied upon by the first minimum processing unit but does not participate in the generation of the final output target field, the first minimum processing unit is retained and only the corresponding target field not involved in the output is removed.

[0033] In one embodiment, the step of traversing the output field of each minimum processing unit in the directed acyclic graph and marking the output field as a redundant field if the output field is not referenced by a subsequent minimum processing unit and is not used as a final output target field includes:

[0034] Traversing each minimum processing unit in the link-level directed acyclic graph, and marking whether its output field is referenced by the next unit or serves as the final output target field;

[0035] If the output field of the minimum processing unit is not referenced and is not used as the final output target field, it is marked as a redundant node;

[0036] Detecting the redundant nodes, and if there are redundant nodes with cross-layer dependencies, recursively tracing back the reference relationship of the output fields of the upstream units of the redundant nodes;

[0037] If the output fields of the upstream unit are all invalid due to downstream pruning, the upstream unit is synchronously marked as the redundant node.

[0038] In one embodiment, after the step of synchronously marking the upstream unit as the redundant node if all output fields of the upstream unit are invalid due to downstream pruning, the method further includes:

[0039] Based on the recursively marked minimum processing unit SDP, all redundant nodes and associated edges are removed layer by layer from the bottom layer to the top layer, and the input-output mapping relationship of the remaining nodes is updated to output the target data processing unit;

[0040] If the field requirements of the input data matrix of the target data processing unit change, the field retention rules of the upstream unit are rechecked, and the steps of traversing each minimum processing unit SDP in the link-level directed acyclic graph and marking whether its output field is referenced by the next unit or serves as the final output target field are iteratively executed until there are no new redundant nodes in the minimum processing unit SDP.

[0041] In addition, to achieve the above-mentioned purpose, the present application also provides a data analysis device based on a directed acyclic graph, and the data analysis device based on a directed acyclic graph includes: a memory, a processor, and a computer program stored on the memory and runnable on the processor, and the computer program is configured to implement the steps of the data link optimization method based on the standard data operation model as described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the steps of the data analysis method based on the directed acyclic graph as described above are implemented.

[0043] The present application provides a data analysis method based on a directed acyclic graph, a data analysis device based on a directed acyclic graph, and a storage medium. The method converts each SQL statement in the data processing job script into a minimum processing unit SDP, and then constructs a SQL-level directed acyclic graph based on the grammatical operation relationship between the minimum processing units. The table dependency relationship between each of the SQL-level directed acyclic graphs is then parsed, and a job-level directed acyclic graph is constructed based on the table dependency relationship. Finally, a link-level directed acyclic graph is constructed based on the job dependency relationship corresponding to the job-level directed acyclic graph, and a data operation model operation is performed on the link-level directed acyclic graph to generate and output an optimized link-level directed acyclic graph. By decomposing a complex data processing process into minimum processing units and constructing a directed acyclic graph, and then performing a data operation model operation on the directed acyclic graph, the accuracy of complex data link analysis and optimization is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 This is a flowchart of a first embodiment of the data analysis method based on a directed acyclic graph of the present application;

[0047] Figure 2 This is a flow chart of the SDP operation involved in the embodiment of this application;

[0048] Figure 3 This is a flow chart of a second embodiment of the data analysis method based on a directed acyclic graph of the present application;

[0049] Figure 4 This is a flowchart of the projection column operation involved in the embodiment of the present application;

[0050] Figure 5 Construct a flowchart for the projection operation relationship involved in the embodiment of this application;

[0051] Figure 6 A flowchart for constructing a selection operation relationship when no subquery is involved in the embodiment of the present application;

[0052] Figure 7 A flowchart for constructing a selection operation relationship involving a subquery in an embodiment of the present application;

[0053] Figure 8 This is a schematic diagram of the union operation flow for a scenario without the ALL flag involved in the embodiment of the present application;

[0054] Figure 9 This is a schematic diagram of the union relation operation flow involving the ALL identification scenario involved in the embodiment of the present application;

[0055] Figure 10 This is a flow chart of the relational algebraic operation of the Cartesian product involved in the embodiment of the present application;

[0056] Figure 11 This is a flow chart of a third embodiment of the data analysis method based on a directed acyclic graph of the present application;

[0057] Figure 12 A flowchart for constructing a job-level directed acyclic graph involved in the embodiment of the present application;

[0058] Figure 13 This is a flowchart of a fourth embodiment of the data analysis method based on a directed acyclic graph of the present application;

[0059] Figure 14 This is a schematic diagram of source data matrix pruning involved in the embodiment of the present application;

[0060] Figure 15 Schematic diagram of the source data matrix pruning scenario involved in the embodiment of the present application;

[0061] Figure 16 This is a schematic diagram of the DAG pruning process with a single SDP as the starting point involved in the embodiment of the present application;

[0062] Figure 17 This is a schematic diagram of a special scenario involved in the embodiment of this application;

[0063] Figure 18 Schematic diagram of the upward movement process of the single source data matrix SDP involved in the embodiment of the present application;

[0064] Figure 19 Schematic diagram of the upward movement process of the multi-source data matrix SDP involved in the embodiment of the present application;

[0065] Figure 20 This is a schematic diagram of the upward migration process starting from a single SDP involved in the embodiment of the present application;

[0066] Figure 21 This is a schematic diagram of the splicing process involved in the embodiment of this application;

[0067] Figure 22 This is a schematic diagram of the link decoupling process involved in the embodiment of the present application;

[0068] Figure 23 This is a schematic diagram of the link shortening process involved in the embodiment of the present application;

[0069] Figure 24 This is a schematic diagram of the architecture of the hardware operating environment of the directed acyclic graph-based data analysis device involved in an embodiment of the present application.

[0070] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0071] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0072] To better understand the above technical solutions, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0073] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0074] The main solution of this application is to convert each SQL statement in the data processing job script into the minimum processing unit SDP;

[0075] Constructing a SQL-level directed acyclic graph based on the grammatical operation relationship between the minimum processing units;

[0076] Parsing table dependencies between each of the SQL-level directed acyclic graphs, and constructing a job-level directed acyclic graph based on the table dependencies;

[0077] Constructing a link-level directed acyclic graph based on the job dependencies corresponding to the job-level directed acyclic graph;

[0078] A data operation model operation is performed on the link-level directed acyclic graph to generate and output an optimized link-level directed acyclic graph.

[0079] Currently, traditional solutions typically extract table-level input-output relationships based on database logs, job scripts, or data catalog tools to generate a cross-job table lineage graph. These solutions then analyze the SELECT field list and JOIN conditions to establish direct reference relationships between fields. However, when faced with massive and complex links, these solutions can only display data dependencies and are unable to analyze the specific processing logic between fields. This results in low accuracy in analyzing and optimizing complex data links.

[0080] This application converts each SQL statement in the data processing job script into the minimum processing unit SDP, and then constructs an SQL-level directed acyclic graph based on the grammatical operation relationship between the minimum processing units, and then parses the table dependency relationship between each of the SQL-level directed acyclic graphs, and constructs a job-level directed acyclic graph based on the table dependency relationship, and finally constructs a link-level directed acyclic graph based on the job dependency relationship corresponding to the job-level directed acyclic graph, and performs data operation model operations on the link-level directed acyclic graph to generate and output an optimized link-level directed acyclic graph. By decomposing the complex data processing process into minimum processing units and constructing a directed acyclic graph, and then performing data operation model operations on the directed acyclic graph, the accuracy of complex data link analysis and optimization is improved.

[0081] Example 1

[0082] Based on this, the embodiment of the present application provides a data analysis method based on a directed acyclic graph, referring to Figure 1 , Figure 1 This is a flow chart of a first embodiment of a data analysis method based on a directed acyclic graph of the present application. The data analysis method based on a directed acyclic graph includes steps S10 to S40:

[0083] Step S10: converting each SQL statement in the data processing script into a minimum processing unit SDP to obtain a data matrix to be processed, performing row and column operations on the data matrix to be processed, and generating a minimum processing unit;

[0084] In this embodiment, the data analysis system performs processing actions. A data processing job script refers to an SQL script file used to execute a series of data processing tasks in a financial data warehouse. It contains multiple SQL statements, each of which implements a specific data processing function, such as data extraction, transformation, and loading. SQL statements are a standard programming language for operating relational databases, enabling operations such as querying, inserting, updating, and deleting data.

[0085] Specifically, the data analysis system retrieves data processing job script files from the data warehouse's job scheduling system. A SQL parser then reads each SQL statement in the script, performing both syntactic and semantic analysis. During the parsing process, key elements of the SQL statement are extracted, such as the field list in the SELECT clause, the table name in the FROM clause, and the conditional expression in the WHERE clause. Based on these key elements, a minimal processing unit (SDP) is constructed, abstracting the SQL statement processing logic into the SDP's input and output relationships and data processing operations.

[0086] Furthermore, to simplify the data operation model, the standard data operation model defines a unified minimum processing unit, SDP (Structured Data Processing). Each SDP is a standard data matrix operation, where a single or double matrix is ​​converted into a new matrix through standard row and column operations.

[0087] Among them, such as Figure 2 As shown, Figure 2 This is the SDP operation flow chart involved in the embodiment of the present application. As shown in the figure, column operations represent the impact of field operations of the source data matrix (M1, M2) on the single column data of the output data matrix (M3); including parsing the Select field expression content and defining field column operations. Row operations represent the impact of field operations of the source data matrix (M1, M2) on the entire row data of the output data matrix (M3); including parsing SQL segments such as join, where, groupby, having, and further abstractly defining multiple standard row operations.

[0088] Step S20: constructing a SQL-level directed acyclic graph according to the grammatical operation relationship between the minimum processing units SDP.

[0089] In this embodiment, the grammatical operation relationship refers to the logical relationship between different data processing operations in an SQL statement. SQL syntax, including SQL syntax, is a specific implementation based on relational algebra that describes the detailed data processing process. The SDP DAG is a simplification of the SQL representation, retaining the most important row and column operation relationships while simplifying various complex expressions. The grammatical operation relationship defines operations such as projection, selection, union, difference, Cartesian product, and aggregation. A SQL-level directed acyclic graph (DAG) is a data structure used to represent the dependencies and execution order between the smallest processing units (SDPs) in an SQL statement. In a data warehouse, SQL statements are used to execute various complex data queries and processing operations.

[0090] Specifically, the grammatical structure of the SQL statement is parsed to identify the various data processing operations (such as the SELECT, JOIN, WHERE clauses of the table) and the grammatical operation relationships between them. Based on these relationships, each data processing operation is converted into a minimum processing unit (SDP), and the connection order between them is determined. When constructing the SQL-level DAG graph, the SDP is used as a node in the graph, and directed edges are added between the nodes based on the grammatical operation relationship to indicate that data flows from one SDP to another. For example, for an SQL statement containing a JOIN operation between two tables, the SELECT operation of the two tables is first converted into two SDP nodes, and then a directed edge is added based on the JOIN condition to indicate that data flows from these two SDP nodes to the SDP node of the JOIN operation. In this process, the system uses the SQL parser to analyze the syntax tree of the SQL statement and extract the various data processing operations and the relationships between them.

[0091] Furthermore, basic data operations are obtained by parsing SQL statements. SQL statements are the core of every data processing task in the data processing chain. By analyzing the grammatical structure of these SQL statements, basic data operations can be identified. For example, parsing the FROM clause in a SELECT statement can determine the data source table, parsing the WHERE clause can determine data filtering conditions, and parsing the GROUP BY clause can determine data grouping methods. Using SQL syntax parsing technology, complex SQL statements can be broken down into a series of basic data operations.

[0092] Step S30: parsing the table dependency relationship between each of the SQL-level directed acyclic graphs, and constructing a job-level directed acyclic graph based on the table dependency relationship.

[0093] In this embodiment, table dependencies refer to the reference relationships between different data tables in a SQL-level directed acyclic graph (DAG). For example, the data in one table depends on the data in another table as input. A job-level DAG is an abstraction of the data processing flow from the perspective of a job. It represents the dependencies between multiple SQL-level DAG graphs. A job typically consists of a sequence of multiple SQL statements. In a data warehouse, a job is the basic unit of data processing, used to achieve specific data processing goals.

[0094] For each SQL-level DAG graph, the data tables involved are parsed to determine the dependencies between these tables and tables in other SQL-level DAG graphs. For example, the output table of one SQL-level DAG graph may serve as the input table of another SQL-level DAG graph. Based on these table dependencies, multiple SQL-level DAG graphs are integrated together to construct a job-level DAG graph. In a job-level DAG graph, nodes represent SQL-level DAG graphs, and directed edges indicate the flow of data from the output of one SQL-level DAG graph to the input of another SQL-level DAG graph. The data processing system traverses the execution order and table dependencies of all SQL statements in the entire job, connecting related SQL-level DAG graphs to form a complete job-level DAG graph.

[0095] As an optional implementation, all basic data operations and their corresponding SDPs in the entire data processing chain can be traversed to determine the output data matrices of the preceding SDPs on which the input data matrix of each SDP depends. Based on these dependencies, corresponding nodes (SDPs) and directed edges are added to the DAG. For example, if the output data matrix of SDPA is one of the input data matrices of SDPB, a directed edge from SDPA to SDPB is added to the DAG. This process analyzes the data flow and constructs a preliminary graph structure that reflects the data processing sequence.

[0096] Furthermore, loops must be checked and eliminated to ensure DAG properties. The initially constructed graph structure must be traversed and checked for loops. If loops are found, the data processing chain must be reviewed and adjusted, potentially splitting or reorganizing certain data processing operations to eliminate the loops. For example, if SDP X depends on the output of SDP Y, and SDP Y depends on the output of SDP X, forming a loop, the data processing logic corresponding to these two SDPs must be reevaluated. This ensures that the final constructed graph structure is a directed acyclic graph.

[0097] Step S40: constructing a link-level directed acyclic graph according to the job dependency relationships corresponding to the job-level directed acyclic graph.

[0098] In this embodiment, job dependencies refer to the execution order dependencies between different jobs in a job-level directed acyclic graph (DAG). For example, the execution of one job requires the output of another job as input. A link-level directed acyclic graph (DAG) is a more macroscopic abstraction of the entire data processing chain. It represents the dependencies between multiple job-level DAG graphs, covering the complete data processing flow from data source to final data product. In financial data warehouses, link-level DAG graphs are used to describe the dependencies between jobs throughout the data warehouse, ensuring the correctness and efficiency of data processing.

[0099] Specifically, pre-set tools are used to parse the dependencies between job-level DAG graphs. For example, if the output table of job A is the input table of job B, then the execution of job B must wait for the completion of job A. Based on these job dependencies, multiple job-level DAG graphs are integrated into a link-level DAG graph. In a link-level DAG graph, nodes represent job-level DAG graphs, and directed edges represent the execution order dependencies between jobs. By traversing the execution order and dependencies of all jobs in the entire data processing chain, the relevant job-level DAG graphs are connected to form a complete link-level DAG graph.

[0100] Step S50: performing a data operation model operation on the link-level directed acyclic graph to generate an optimized link-level directed acyclic graph.

[0101] In this embodiment, data computation model operations are a general term for a series of optimization operations performed on the smallest processing units (SDPs) and their connections in a directed acyclic graph (DAG). Data computation model operations include pruning, moving up, and splicing. Pruning refers to removing unnecessary data processing branches or redundant fields based on user needs or data usage; moving up refers to moving certain SDPs forward in the DAG to reduce data processing steps or improve data processing timeliness; and splicing refers to merging or recombining multiple SDPs to generate more concise and efficient data processing logic.

[0102] Specifically, the link-level directed acyclic graph is first traversed and analyzed to identify nodes and edges that can be optimized. For example, by examining the data dependencies and operation types between nodes, pairs of nodes that can be merged are identified. For example, two consecutive processing units that perform simple operations on the same field can be merged into a single composite operation unit. Furthermore, data flow is analyzed to remove redundant edges that do not change the data's content or are already covered by other paths. This results in the generation of an optimized target data processing link.

[0103] In the technical solution provided in this embodiment, the SQL statements in the data processing job script are converted into minimum processing units SDP one by one, and then a SQL-level directed acyclic graph is constructed based on the grammatical operation relationship between the minimum processing units. Then, the table dependency relationship between each of the SQL-level directed acyclic graphs is parsed, and a job-level directed acyclic graph is constructed based on the table dependency relationship. Finally, a link-level directed acyclic graph is constructed based on the job dependency relationship corresponding to the job-level directed acyclic graph, and the data operation model operation is performed on the link-level directed acyclic graph to generate and output the optimized link-level directed acyclic graph. This embodiment improves the accuracy of complex data link analysis and optimization by decomposing complex data processing processes into minimum processing units and constructing a directed acyclic graph, and then performing data operation model operations on the directed acyclic graph. And by decomposing complex data processing processes into minimum processing units and constructing a directed acyclic graph, the data processing logic is made clearer and more intuitive, which is convenient for data developers to understand and maintain. On the other hand, by performing data calculation model operations such as pruning, moving up, and splicing, redundant data processing steps and fields are eliminated, the waste of system resources is reduced, and the efficiency and performance of data processing are improved.

[0104] In addition, similar technologies establish a kinship relationship between the source and target tables and filter the source table fields on the target table, but do not identify or process whether the relationship can be trimmed. Furthermore, there is a lack of good visualization tools for representing such relationships. Once the link level is deep, for example, 50 layers, it is impossible to analyze whether the table fields on the first layer affect the fields on the 50th layer. This embodiment decomposes the complex data processing flow into minimum processing units and constructs a directed acyclic graph, thereby enabling in-depth analysis of the link.

[0105] Example 2

[0106] Based on the same inventive concept, this application also provides a second embodiment, please refer to Figure 3 , Figure 3 This is a flow chart of the second embodiment of the data analysis method based on a directed acyclic graph of the present application, wherein step S20 includes steps S21 to S23:

[0107] Step S21: when parsing the minimum processing unit SDP, if the minimum processing unit SDP contains subqueries, the subqueries are independently decomposed into nested basic data operation sets, and mapped layer by layer to the corresponding minimum processing units.

[0108] Step S22: using the output matrix of the minimum processing unit SDP corresponding to the sub-query as the input matrix of the main query.

[0109] Step S23: constructing a SQL-level directed acyclic graph including sub-query data processing units and main query data processing units according to the basic data operations of the main query, wherein the sub-query data processing units are embedded in the hierarchical structure of the main query as independent nodes.

[0110] In this embodiment, a subquery refers to another SQL query nested within an SQL query (the main query). It is used to generate intermediate result sets required by the main query, allowing the main query to perform further data processing based on these intermediate results. For example, in a data warehouse, the main query searches for customer information whose amounts are greater than the average amount calculated by a subquery. Here, the subquery is responsible for calculating the average transaction amount. The output matrix refers to the new data set generated after data processing operations on the minimum processing unit (SDP) corresponding to the subquery. It is represented in a two-dimensional table format and contains field values ​​calculated according to the subquery logic. The input matrix of the main query refers to the data set required by the main query statement. It is usually provided by the original data table or a data table generated through other data processing operations and is used for further data processing of the main query. A SQL-level directed acyclic graph (DAG) is a DAG graph constructed specifically for the data processing logic in SQL query statements. It uses the minimum processing unit (SDP) as a node, and directed edges represent the flow of data between SDPs. The entire graph does not contain loops. The main query data processing unit refers to the SDP converted from the basic data operations in the main query statement. The subquery data processing unit is embedded in the hierarchical structure of the main query as an independent node. That is, when constructing the DAG graph, the SDP corresponding to the subquery is regarded as an independent node associated with the main query SDP, and is correctly placed in the DAG graph hierarchy corresponding to the main query according to the order and dependency of data processing.

[0111] Specifically, when parsing the SQL statement in the data processing link, the existence of the subquery is identified by the SQL syntax parser. Then, the subquery is independently disassembled into a nested set of basic data operations. The grammatical structure of the subquery is then analyzed layer by layer to determine the basic data operations such as table references, field selections, and data screening conditions. Next, these basic data operations are mapped layer by layer to the corresponding minimum processing unit (SDP). For example, if the subquery contains an operation that groups the transaction table by customer and calculates the total transaction amount, then it is mapped to an SDP, with the input data matrix being the transaction table and the output data matrix being the customer-total transaction amount table, in which the field-level operation logic is defined, that is, the transaction amount field of each customer is summed.

[0112] After converting the subquery into an SDP and determining its output matrix, this output matrix needs to serve as the input matrix for the main query. Specifically, when constructing the directed acyclic graph (DAG) of the data processing chain, the output data matrix of the subquery SDP is connected to the input of the SDP corresponding to the main query through data flow. This allows the data generated by the subquery SDP to be retrieved when the data processing flow reaches the SDP corresponding to the main query, serving as the basis for its data processing. The structure and logic of the corresponding SDP are then determined based on the basic data operations of the main query. Then, when constructing the SQL-level DAG, the SDP node corresponding to the subquery is added to the graph, and the connection between it and the main query SDP node is determined based on data dependencies. Specifically, the subquery SDP node is placed at a level before the main query SDP node because the main query requires the subquery's results as input. In the DAG, the output of the subquery SDP is connected to the input of the main query SDP via directed edges, accurately reflecting the data processing flow.

[0113] Optionally, when constructing a SQL-level directed acyclic graph, it's necessary to obtain the relational algebra of the data processing chain. SQL syntax itself is a specific implementation of relational algebra, describing the detailed data processing process. SDP's DAG is a simplified SQL representation, preserving the most important row-column operations while simplifying various complex expressions. Relational algebra defines operations such as projection, selection, union, difference, Cartesian product, and aggregation. By figuring out how to convert these operations into SDP, the SQL-to-SDP conversion problem is solved.

[0114] Specifically, if Figure 4 As shown, Figure 4 This is a flowchart of the projection column operation involved in the embodiment of the present application. First, obtain the projection in the relational algebra. The projection is a column operation in the relational algebra, which corresponds to the Select clause part "Select expression 1, expression 2, ..., expression N" in SQL. Each expression contains 0 to multiple fields in the source table. If these expressions do not contain subqueries, the relationship between the source data matrix and the target data matrix can be constructed based on the relationship between the source fields and the expressions. If there are subqueries in the expression, the construction process is different. Figure 5 As shown, Figure 5 A flowchart is constructed for the projection operation relationship involved in the embodiment of this application, such as Figure 5 Assuming that expression 1 contains a subquery (the processing method for multiple subqueries is similar), the construction process needs to first convert the subquery into an SDP, then build a new SDP based on the processing method without subqueries, and finally build the corresponding column matrix by combining the subquery SDP and the target data matrix.

[0115] For the relational algebra operation of selection, selection is the filtering of the data in the set in relational algebra, which corresponds to the "where expression" and "having expression" in SQL. In SDP, it is a typical row operation. If there is no subquery in the expression, the process of building SDP is as follows: Figure 6 As shown, Figure 6 This is a flowchart for constructing a selection operation relationship when there is no subquery involved in the embodiment of this application. By extracting the expression corresponding to the selection in the source data matrix and performing row operations, and performing column operations on other fields, the output matrix is ​​determined. If there is a subquery in the expression (multiple subqueries are similar), such as Figure 7 As shown, Figure 7 This is a flowchart for constructing the selection operation relationship when a subquery is included in the embodiment of the present application. The construction process needs to first convert the subquery into an SDP, and then construct a new SDP with reference to the processing method without a subquery, and finally construct a corresponding column matrix of the subquery SDP and the target data matrix.

[0116] The relational algebra operation of "and" is to merge the data in two sets, which corresponds to UNION[ALL] in SQL syntax. ALL indicates whether duplicate data in the two sets needs to be retained. The SDP construction process includes two cases depending on whether the ALL flag is included. In the scenario without the ALL flag, such as Figure 8 As shown, Figure 8 This is a schematic diagram of the process flow of the union operation of the embodiment of the present application without the ALL flag scenario. As shown in the figure, all fields of the source data matrix generate row operations on the target data matrix. For the scenario containing the ALL flag, Figure 9 As shown, Figure 9 This is a schematic diagram of the union relation operation flow involving the ALL identification scenario involved in the embodiment of the present application. There is no need to perform row operations, and the two source data matrices being operated can be subjected to column operations.

[0117] The relational algebraic operation of difference is a set difference operation between two sets A and B. The result is to retain the data in A that is not in set B. The structure of the generated SDP is the same as that of UNION without the ALL flag, so it will not be repeated here.

[0118] For the relational algebra operation of Cartesian product, Cartesian product is a connection operation between two sets in relational algebra, which corresponds to the JOIN clause in SQL. Figure 10 As shown, Figure 10This is a flow chart of the relational algebra operations of the Cartesian product involved in the embodiment of the present application. For multi-table associations, according to the order of association, it can be converted into an association between two tables. For example, AJOIN B JOIN C can be changed to (A JOIN B) JOIN C. Usually, the association between tables is limited by the "ON expression" clause. If the "ON expression" contains a subquery, A JOIN B can be converted to AJOIN (B JOIN subquery). Therefore, converting the Cartesian product to SDP only requires considering the association between the two tables.

[0119] Aggregation is a special type of set operation that corresponds to the group by clause in SQL and uses row operations in SDP. The resulting SDP structure is the same as that generated by the selected relational operation.

[0120] Furthermore, in this embodiment, after step 33, the following steps are further included:

[0121] Determine the target table dependency between the data processing links; based on the target table dependency, merge the directed acyclic graph into a job-level directed acyclic graph, wherein the nodes of the job-level directed acyclic graph represent a set of minimum processing units, and the edges represent the data flow relationship across data processing statements.

[0122] Specifically, the target table dependency is the dependency between tables in the data processing link. Through the target table dependency, the directed acyclic graph can be merged into a job-level directed acyclic graph. The nodes of the job-level directed acyclic graph represent the set of minimum processing units, and the edges represent the data flow relationship across data processing statements.

[0123] In the technical solution provided in this embodiment, by independently decomposing and mapping the subqueries in the data processing link into minimum processing units and rationally constructing a SQL-level directed acyclic graph, effective processing and optimization of the subquery part in the complex SQL query statement is achieved. On the one hand, the subquery is converted into an independent SDP so that it can participate in the optimization process of the entire data processing link like an ordinary data processing operation, which helps to discover and eliminate redundant calculations or unnecessary data processing steps that may exist in the subquery. On the other hand, the subquery SDP is correctly embedded in the hierarchical structure of the main query, which ensures the logical correctness and sequentiality of the data processing, and at the same time provides a clearer and more accurate model foundation for subsequent large-scale link decoupling analysis and other work, further improving the efficiency and quality of data link optimization in the financial data warehouse, and better meeting the strict requirements of financial data processing for accuracy and efficiency.

[0124] Example 3

[0125] Based on the same inventive concept, this application also provides a third embodiment, please refer to Figure 11 , Figure 11 This is a flow chart of the third embodiment of the data analysis method based on a directed acyclic graph of the present application, wherein step S30 includes steps S31 to S34:

[0126] Step S31: parsing the table dependency relationship of each SQL-level directed acyclic graph;

[0127] Step S32: Among the statement pairs with the data dependency relationship, the output data matrix of the minimum processing unit SDP corresponding to the upstream statement is used as the input data matrix of the unit corresponding to the downstream statement;

[0128] Step S33: determining an input-output relationship between the output data matrix and the input matrix, and connecting at least one of the SQL-level directed acyclic graphs into a job-level directed acyclic graph according to the input-output relationship;

[0129] Step S34: If there are independent data processing statements without dependencies in the data processing link, the corresponding target SQL-level directed acyclic graph is retained and used as a subgraph of the job-level directed acyclic graph.

[0130] In this embodiment, when constructing the job-level directed acyclic graph, at least one SQL in the data processing link is first parsed to obtain the data dependency relationship between the target table and the source table. Then, in the statement pair with the data dependency relationship, the output data matrix of the minimum processing unit corresponding to the upstream statement is used as the input data matrix of the unit corresponding to the downstream statement; the input-output relationship between the output data matrix and the input matrix is ​​determined, and at least one of the SQL-level directed acyclic graphs is connected to form a job-level directed acyclic graph based on the input-output relationship. If there is an independent data processing statement without a dependency relationship in the data processing link, the corresponding target SQL-level directed acyclic graph is retained and used as a subgraph of the job-level directed acyclic graph.

[0131] Optionally, in this embodiment, step S31 includes:

[0132] Traverse the minimum processing unit list in the SQL-level directed acyclic graph; construct a calculation model of the SQL-level directed acyclic graph based on the minimum processing unit list; obtain a source table in the SQL-level directed acyclic graph, and query the target calculation model corresponding to the source table; construct a parent-child relationship of model nodes based on the target calculation model, and determine the parent-child relationship as the table dependency.

[0133] Specifically, if Figure 12 As shown, Figure 12This is a flowchart for constructing a job-level directed acyclic graph involved in the embodiment of the present application. Usually, the processing program of a job contains at least one SQL, and each SQL can be converted into a DAG graph of SDP according to the above method. Assuming that there are two SQLs (SQL1 and SQL2) in a job, DAG graphs of different job-level SDPs can be generated based on the relationship between SQL1 and SQL2. If the target table of SQL1 is not the source table of SQL2, SQL1 is unrelated to SQL2, and therefore there is no need to establish a connection between the DAG graphs of the two. If the target table of SQL1 is the source table of SQL2, then a connection is established.

[0134] In addition, the DAG graph construction of the link-level SDP is consistent with the construction of the job-level SDP. Connections are established between the DAG graphs of the job-level SDP based on the dependencies of the jobs and the relationships between the source and target tables in the DAG graph of the job-level SDP.

[0135] The technical solution provided in this embodiment achieves refined management and optimization of complex data processing flows by deeply analyzing the table dependencies of the SQL-level directed acyclic graph and constructing a job-level directed acyclic graph based on these dependencies. This improves the accuracy and efficiency of data processing by ensuring the correct transfer and processing of data between different SQL statements. Furthermore, the job-level directed acyclic graph can better organize and manage various data processing tasks for the processing of independent data processing statements without dependencies, improving the maintainability and scalability of the jobs.

[0136] Example 4

[0137] Based on the same inventive concept, this application also provides a fourth embodiment, please refer to Figure 13 , Figure 13 This is a flow chart of a fourth embodiment of the data analysis method based on a directed acyclic graph of the present application, wherein step S30 includes steps S60 to S80:

[0138] Step S60: traverse the output field of each minimum processing unit SDP in the link-level directed acyclic graph, and mark the output field as a redundant field if the output field is not referenced by a subsequent minimum processing unit SDP and is not used as a final output target field.

[0139] Step S70: If all output fields of the target minimum processing unit are marked as redundant fields, the target minimum processing unit is marked as a redundant minimum processing unit.

[0140] Step S80: Based on the dependency path of the output fields in the link-level directed acyclic graph, remove redundant fields and / or redundant minimum processing units that do not participate in the final output to complete the data calculation model operation on the directed acyclic graph.

[0141] In this embodiment, the output field is the data generated after the minimum processing unit performs a data processing operation. Redundant fields refer to output fields that are neither used by subsequent processing units nor the final analysis target in the data processing flow. The existence of such fields increases data storage costs and processing time, reducing data processing efficiency. A redundant minimum processing unit refers to a processing unit in the data link whose all output fields are determined to be redundant fields. In other words, the data processing operations performed by this processing unit have no impact on the final data analysis results, and its existence only increases the complexity of data processing and resource consumption. The output field dependency path is the transmission and usage relationship between different processing units in the data link, reflecting the complete flow path of data from the source to the final target. Along this path, data can undergo multiple processing and transformations, and each processing unit may operate on certain fields or generate new fields.

[0142] Specifically, when performing data link optimization, starting from the starting node of the directed acyclic graph, each minimum processing unit is visited in turn. For each processing unit, its output field list is checked and compared with the input field requirements of all subsequent minimum processing units. At the same time, the final output target field definition file is compared to determine whether each output field is a required field. Specifically, a field reference relationship table can be constructed to record which subsequent processing units reference each output field. For example, for a complex data link containing multiple data conversion and calculation steps, the field name generated by each intermediate processing step and its corresponding downstream usage are recorded. If the detection module detects that a certain output field does not appear in the input field list of any subsequent processing unit or in the final output target field set, it will be marked as a redundant field according to the preset rules.

[0143] After marking all minimum processing unit output fields for redundancy, the system examines the output field list of each processing unit and counts the number of fields marked as redundant. If all output fields of a processing unit are marked as redundant, the processing unit is marked as a redundant minimum processing unit according to predefined redundancy handling rules. For example, in a data chain consisting of multiple data cleansing, transformation, and aggregation steps, the system examines the output fields of each step one by one. Steps whose output fields are not used by subsequent steps or the final report are marked as redundant processing units. After marking redundant fields and redundant processing units, the system uses the field dependency path information to formulate a removal strategy. Starting from the terminal node of the data chain, the entire directed acyclic graph is traversed in reverse, tracing back along the output field dependency path. For each field marked as redundant, the system examines the minimum processing unit in which it resides and any other processing units that reference it. If a field is used only in a redundant processing unit, it can be removed along with the redundant processing unit. However, if a field is referenced by multiple processing units, the system analyzes their dependencies to ensure that its removal does not affect the data needs of other normal processing units.

[0144] Optionally, in this embodiment, step S50 includes:

[0145] Traverse each minimum processing unit in the link-level directed acyclic graph and mark whether its output field is referenced by the next unit or serves as the final output target field; if the output field of the minimum processing unit is not referenced and is not used as the final output target field, mark the target unit as a redundant node; detect the redundant node, and if there is a redundant node with cross-layer dependencies, recursively trace back the reference relationship of the upstream unit output field of the redundant node; if the output fields of the upstream unit are all invalid due to downstream pruning, synchronously mark the upstream unit as the redundant node.

[0146] Optionally, the above operations include pruning operations, moving up operations, and splicing operations.

[0147] For pruning operations, it can be applied to source data matrix field pruning, source data matrix pruning, and single SDP-based DAG pruning. In the source data matrix field pruning scenario, such as Figure 14 As shown, Figure 14 This is a schematic diagram of source data matrix pruning involved in the embodiment of the present application. If certain fields in the SDP do not participate in row calculations or column calculations of reserved fields, these fields in Table A can be pruned. For example, if the C2 and C3 fields of M1 do not participate in row and column calculations, and the C4 field of M2 also performs row and column calculations on the C1, C2, and C3 reserved fields of M3, these fields can be pruned.

[0148] In the source data matrix pruning scenario, if the row operation in the SDP is only a LEFT JOIN, the fields of the source data matrix do not participate in the column operation, and the cardinality of the result set is not changed, then the source data matrix can be pruned. Figure 15 As shown, Figure 15 This is a schematic diagram of the source data matrix pruning scenario involved in the embodiment of the present application, such as Figure 15 ,If the fields in M2 do not participate in the column operation, then M2 in SDP is pruned.

[0149] In the scenario of DAG pruning with a single SDP as the starting point, DAG pruning is essentially based on a graph traversal algorithm combined with a single SDP pruning method, outputting which edges in the DAG can be pruned. However, unlike ordinary graph traversal methods, the SDP in the graph can only be pruned after all sub-SDPs have been pruned. Figure 16 For example, Figure 16 This is a schematic diagram of the DAG pruning process with a single SDP as the starting point involved in the embodiment of the present application. SDP1 must be pruned after SDP2 and SDP3 are pruned, based on the union of the reserved fields of SDP1 required by SDP2 and SDP3.

[0150] In addition, during the pruning process, after step S50, the following steps are further included:

[0151] If the output data matrices of two minimum processing units are the same data object, the two units are merged into a single node and their input-output dependency is updated; when removing redundant minimum processing units, if the output data matrix of the redundant minimum processing unit is dependent on the first minimum processing unit but does not participate in the generation of the final output target field, the first minimum processing unit is retained and only the corresponding target field that does not participate in the output is removed.

[0152] Specifically, in the above-mentioned cutting process, there are some special scenarios, such as Figure 17 As shown, Figure 17 Schematic diagram of a special scenario involved in the embodiment of the present application. If M1 and M3 correspond to the same object, then when pruning SDP1, since all four fields of M3 need to be retained, col4 of M1 needs to be retained and cannot be pruned. In the following scenario, T1 and M2 are the same object, and col3 of T1 participates in the non-pruneable row operation of M1. When pruning SDP3, M2 only needs to retain two fields. When pruning SDP2, col3 is pruned, which does not meet the requirements of SDP1. Therefore, in actual operation, when pruning to SDP1, when T1 adds a new reserved field, SDP2 and SDP1 need to be pruned again.

[0153] Optionally, the upward operation specifically includes the upward movement of a single source data matrix SDP, the upward movement of a multi-source data matrix SDP, and the upward movement process starting from a single SDP. Figure 18 As shown, Figure 18 This is a schematic diagram of the upward movement process of the single source data matrix SDP involved in the embodiment of the present application. The upward movement of the single source data matrix SDP is that the lower layer SDP can skip M3 and directly use M1. Figure 19 As shown, Figure 19 This is a schematic diagram of the upward migration process of the multi-source data matrix SDP involved in the embodiment of the present application. In the multi-source data matrix SDP, if M2 does not participate in the column operation and only participates in the LEFT JOIN row operation, then the SDP can be moved upward. The lower layer SDP can skip M3 and use M1 directly. The upward migration process starting from a single SDP is as follows: Figure 20 For example, Figure 20 This is a schematic diagram of the upward migration process starting from a single SDP involved in the embodiment of the present application. After pruning, T2 is pruned, and SDP1 is the SDP of the single source data matrix and can be moved upward. After the upward analysis, SDP2 no longer depends on M1, but on T1. Splicing is to convert the standard data operation model back to SQL, which is used to help users rewrite the optimized job script after pruning or moving upward. Figure 21 As shown, Figure 21 This is a schematic diagram of the splicing process involved in the embodiment of the present application. The optimized target SQL is generated by splicing the original SQL with the pruned or moved-up SQL. The original SQL is selected as the input because different SQLs can correspond to the same standard data operation model. The SQL directly converted back based on the standard data model is quite different from the original SQL, and it is difficult for users to judge the accuracy of the results. The original SQL syntax tree is rewritten based on the pruned result, and the rewritten SQL basically retains the structure of the original SQL.

[0154] Furthermore, in this embodiment, after the step of synchronously marking the upstream unit as the redundant node if all the upstream unit output fields are invalid due to downstream pruning, the method further includes:

[0155] Based on the recursively marked minimum processing unit, all redundant nodes and associated edges are removed layer by layer from the bottom layer to the top layer, and the input-output mapping relationship of the remaining nodes is updated to output the target data processing unit; if the field requirement of the input data matrix of the target data processing unit changes, the field retention rule of the upstream unit is rechecked, and the step of traversing each minimum processing unit in the link-level directed acyclic graph and marking whether its output field is referenced by the next unit or used as the final output target field is iterated until there are no new redundant nodes in the minimum processing unit.

[0156] In the technical solution provided in this embodiment, by performing specific steps of data calculation model operations on directed acyclic graphs, accurate identification and removal of redundant fields and redundant processing units in data links are achieved, effectively reducing data storage requirements and computing resource consumption, and significantly improving the efficiency and performance of data processing processes, making data links more concise and efficient, and easy to maintain and expand.

[0157] Example 5

[0158] Based on the same inventive concept, the present application also provides a fifth embodiment:

[0159] In this embodiment, the directed acyclic graph-based data analysis method of the present application can be applied to scenarios of link decoupling, link shortening, code purification, and model purification.

[0160] In the link decoupling scenario, based on the standard data operation model of the data warehouse, starting from the SDP of any terminal job, the SDP of any terminal job set, or the SDP of all terminal jobs in the data warehouse, the entire link processing job of the data warehouse is scanned and analyzed from the bottom up, breaking through the illusion of "field reference is use" and "job dependency is useful" under the traditional data lineage relationship, discovering the opportunity points for decoupling of job link dependencies and promoting optimized governance.

[0161] Taking the example of Job 1 and Job 2 whose dependencies can be decoupled and disconnected, the tailoring analysis process is as follows: Figure 22 As shown, Figure 22 This is a schematic diagram of the link decoupling process involved in the embodiment of the present application. Starting from the target table T1 of job 1, upward pruning analysis is performed. Through the pruning operation on the DAG graph, the target table T8 generated by job 2 is referenced in job 1, but can be pruned based on the following logic (this is one of the logics):

[0162] (1) The C3 field of T8 participates in the column operation to generate the C3 of M9. However, the C3 field of M9 is not used in the subsequent process and can actually be trimmed.

[0163] (2) Although the C1 field of T8 is used for row operations together with the C1 field of T7, it can be trimmed and the row operations can be performed using the inheritance mode of Left Join.

[0164] Through link decoupling analysis, unnecessary job dependencies are eliminated, reducing the impact of changes and anomalies in upstream job links on downstream starting jobs or job sets. This purifies and streamlines high-security job links, and even the entire job chain, streamlining assets, improving data timeliness, and reducing data management and assurance costs. This analysis technology can be used not only for bottom-up, full-link decoupling analysis, but also for looking downstream from the source, or by specifying any source and target job nodes to identify key link nodes where the source and target jobs can be decoupled, enabling rapid decoupling analysis of the source and target.

[0165] In the scenario of link shortening, based on the data operation model, by analyzing the field processing path in the data processing flow, the similarity and redundant links between two tables / fields are identified, thereby optimizing the data flow and reducing unnecessary intermediate processing steps. Specifically, it includes:

[0166] 1. Analyze the field processing path: Through full-link analysis of the data processing process, identify the processing path of each field and clarify the flow of data from the source to the target table.

[0167] 2. Identify redundant links: During the analysis process, it was found that the processing logic between certain tables was highly similar, or that some intermediate links had no substantial impact on the final results, thus identifying redundant links.

[0168] 3. Optimize data flow: By identifying redundant links, it is recommended to skip unnecessary intermediate tables or processing steps and directly use upstream tables or data sources, thereby shortening the data processing chain.

[0169] For example, Figure 23 As shown, Figure 23 This is a schematic diagram of the link shortening process involved in the embodiment of this application. M1 is processed from T1 without any complex logic in between. For the target table M3, the processing of M1 does not introduce new logic or field changes, so M1 can be skipped and data can be directly obtained from T1. This link shortening method can significantly reduce the intermediate steps in data processing, reduce resource consumption, and improve data processing efficiency.

[0170] In code purification scenarios, based on a standard data operation model, comprehensive pruning and scanning analysis is performed on the DAG graph of each Gaussian core warehouse processing job script, accurately identifying redundant code in the job script, such as redundant SQL code, redundant fields, and redundant tables. The code is also automatically rewritten to generate purified code, assisting data development engineers in efficiently completing purification and governance tasks. This code purification capability improves code readability, maintainability, and operational efficiency, while reducing unnecessary upstream job dependencies.

[0171] In model purification scenarios, based on standard data computation models, we conduct in-depth analysis of table and field processing to identify suspected redundant models, promote their management, and achieve model optimization and streamlining. Suspected redundant models primarily include tables and fields that are not accessed downstream. These models are referenced by downstream code, but full-link pruning analysis reveals that they can actually be pruned. Similar model analysis reveals that two tables / fields share the same or similar processing logic.

[0172] The present application provides a data analysis device based on a directed acyclic graph, and the data analysis device based on a directed acyclic graph includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data analysis method based on the directed acyclic graph in the above-mentioned embodiment one.

[0173] Reference below Figure 24 , which shows a schematic structural diagram of a directed acyclic graph-based data analysis device suitable for implementing an embodiment of the present application. The directed acyclic graph-based data analysis device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 24 The data analysis device based on a directed acyclic graph shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0174] like Figure 24As shown, the data analysis device based on the directed acyclic graph may include a processing device 1001 (such as a core processor, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM) 1004. In the random access memory 1004, various programs and data required for the operation of the data analysis device based on the directed acyclic graph are also stored. The processing device 1001, the read-only memory 1002 and the random access memory 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data analysis device based on the directed acyclic graph to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a data analysis device based on a directed acyclic graph with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0175] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0176] The directed acyclic graph-based data analysis device provided by the present application adopts the directed acyclic graph-based data analysis method in the above-mentioned embodiment, which can solve the technical problem of low accuracy of traditional solutions in optimizing complex data link analysis. Compared with the prior art, the beneficial effects of the directed acyclic graph-based data analysis device provided by the present application are the same as the beneficial effects of the directed acyclic graph-based data analysis method provided by the above-mentioned embodiment, and the other technical features of the directed acyclic graph-based data analysis device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0177] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0178] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0179] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the directed acyclic graph-based data analysis method in the above-mentioned embodiment.

[0180] The computer-readable storage medium provided in this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM, CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequency (RF, Radio Frequency), etc., or any suitable combination thereof.

[0181] The computer-readable storage medium may be included in a data analysis device based on a directed acyclic graph; or may exist independently without being assembled into a data analysis device based on a directed acyclic graph.

[0182] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a data analysis device based on a directed acyclic graph, the data analysis device based on the directed acyclic graph: converts each SQL statement in the data processing job script into a minimum processing unit SDP; constructs an SQL-level directed acyclic graph based on the grammatical operation relationship between the minimum processing units; parses the table dependency relationship between each of the SQL-level directed acyclic graphs, and constructs a job-level directed acyclic graph based on the table dependency relationship; constructs a link-level directed acyclic graph based on the job dependency relationship corresponding to the job-level directed acyclic graph; performs data operation model operations on the link-level directed acyclic graph, and generates and outputs an optimized link-level directed acyclic graph.

[0183] The computer program code for performing the operations of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0184] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0185] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0186] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned directed acyclic graph-based data analysis method. This computer-readable storage medium can address the technical issue of low accuracy in traditional approaches to analyzing and optimizing complex data links. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the directed acyclic graph-based data analysis method provided in the aforementioned embodiments, and are not further elaborated here.

[0187] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned directed acyclic graph-based data analysis method.

[0188] The computer program product provided in this application can address the technical issue of low accuracy in traditional approaches to analyzing and optimizing complex data links. Compared to the prior art, the beneficial effects of the computer program product provided in this embodiment are similar to those of the directed acyclic graph-based data analysis method provided in the aforementioned embodiment, and are not further elaborated here.

[0189] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A data analysis method based on a directed acyclic graph, characterized in that: The data analysis method based on directed acyclic graph comprises the following steps: Convert each SQL statement in the data processing script into the minimum processing unit SDP; Constructing a SQL-level directed acyclic graph according to the grammatical operation relationship between the minimum processing units SDP; Parsing table dependencies between each of the SQL-level directed acyclic graphs, and constructing a job-level directed acyclic graph based on the table dependencies; Constructing a link-level directed acyclic graph based on the job dependencies corresponding to the job-level directed acyclic graph; A data operation model operation is performed on the link-level directed acyclic graph to generate and output an optimized link-level directed acyclic graph.

2. The method according to claim 1, wherein The step of constructing an SQL-level directed acyclic graph according to the grammatical operation relationship between the minimum processing units SDP includes: When parsing the minimum processing unit SDP, if the minimum processing unit SDP contains subqueries, the subqueries are independently decomposed into nested basic data operation sets, and mapped layer by layer to the corresponding minimum processing units; The output matrix of the minimum processing unit SDP corresponding to the sub-query is used as the input matrix of the main query; According to the basic data operations of the main query, a SQL-level directed acyclic graph including sub-query data processing units and main query data processing units is constructed, wherein the sub-query data processing units are embedded in the hierarchical structure of the main query as independent nodes.

3. The method according to claim 1, wherein The step of parsing the table dependency relationship between each of the SQL-level directed acyclic graphs and constructing the job-level directed acyclic graph based on the table dependency relationship further includes: parsing the table dependencies of each of the SQL-level directed acyclic graphs; In the statement pairs with the table dependency relationship, the output data matrix of the minimum data processing unit SDP corresponding to the upstream statement is used as the input data matrix of the unit corresponding to the downstream statement; Determining an input-output relationship between the output data matrix and the input matrix, and connecting at least one of the SQL-level directed acyclic graphs into a job-level directed acyclic graph according to the input-output relationship; If there are independent data processing statements without dependencies in the data processing link, the corresponding target SQL-level directed acyclic graph is retained and used as a subgraph of the job-level directed acyclic graph.

4. The method according to claim 3, wherein The step of parsing the table dependency relationship of each SQL-level directed acyclic graph includes: Traversing a list of minimum processing units in the SQL-level directed acyclic graph; Constructing a computational model of the SQL-level directed acyclic graph according to the minimum processing unit list; Obtain a source table in the SQL-level directed acyclic graph, and query a target computing model corresponding to the source table; A parent-child relationship of model nodes is constructed based on the target operation model, and the parent-child relationship is determined as the table dependency relationship.

5. The method according to claim 1, wherein The step of performing a data operation model operation on the link-level directed acyclic graph to generate and output an optimized link-level directed acyclic graph includes: Traversing the output field of each minimum processing unit SDP in the link-level directed acyclic graph, and marking the output field as a redundant field if the output field is not referenced by a subsequent minimum processing unit SDP and is not used as a final output target field; If all output fields of the target minimum processing unit are marked as redundant fields, the target minimum processing unit is marked as a redundant minimum processing unit; Based on the dependency path of the output fields in the link-level directed acyclic graph, redundant fields and / or redundant minimum processing units that do not participate in the final output are removed to complete the data calculation model operation on the directed acyclic graph.

6. The method according to claim 5, wherein After the step of marking the target minimum processing unit as a redundant minimum processing unit if all output fields of the target minimum processing unit are marked as redundant fields, the method further includes: If the output data matrices of two minimum processing units SDP are the same data object, the two minimum processing units SDP are merged into a single node, and their input-output dependency is updated; When removing redundant minimum processing units, if the output data matrix of the redundant minimum processing unit is relied upon by the first minimum processing unit but does not participate in the generation of the final output target field, the first minimum processing unit is retained and only the corresponding target field not involved in the output is removed.

7. The method according to claim 5, wherein The step of traversing the output field of each minimum processing unit in the directed acyclic graph and marking the output field as a redundant field if the output field is not referenced by a subsequent minimum processing unit and is not used as a final output target field comprises: Traversing each minimum processing unit in the link-level directed acyclic graph, and marking whether its output field is referenced by the next unit or serves as the final output target field; If the output field of the minimum processing unit is not referenced and is not used as the final output target field, it is marked as a redundant node; Detecting the redundant nodes, and if there are redundant nodes with cross-layer dependencies, recursively tracing back the reference relationship of the output fields of the upstream units of the redundant nodes; If the output fields of the upstream unit are all invalid due to downstream pruning, the upstream unit is synchronously marked as the redundant node.

8. The method according to claim 7, wherein After the step of synchronously marking the upstream unit as the redundant node if all the output fields of the upstream unit are invalid due to downstream pruning, the method further includes: Based on the recursively marked minimum processing unit SDP, all redundant nodes and associated edges are removed layer by layer from the bottom layer to the top layer, and the input-output mapping relationship of the remaining nodes is updated to output the target data processing unit; If the field requirements of the input data matrix of the target data processing unit change, the field retention rules of the upstream unit are rechecked, and the steps of traversing each minimum processing unit SDP in the link-level directed acyclic graph and marking whether its output field is referenced by the next unit or serves as the final output target field are iteratively executed until there are no new redundant nodes in the minimum processing unit SDP.

9. A data analysis device based on a directed acyclic graph, characterized in that: The data analysis device based on a directed acyclic graph includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data link optimization method based on a standard data operation model as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the directed acyclic graph-based data analysis method according to any one of claims 1 to 8 are implemented.

Images