Metadata blood relationship automatic generation tool based on grammar rules

Abstract

The invention discloses an automatic metadata consanguinity generation tool based on grammar rules, and relates to the field of data consanguinity analysis, and the automatic metadata consanguinity generation tool comprises the following steps: S1, automatically collecting a data processing process through an interface, and collecting a unique ID and a version number of each processing task at the same time; the method comprises the following steps: S1, storing a database in each data processing task, storing a timestamp of the database in each data processing task, and reserving each version of data of the task, S3, blood relationship analysis: analyzing SQL statements of the processing data, and converting the SQL statements into instructions which can be understood and executed by a computer, and S4, visualizing blood relationship data through a blood relationship analysis criterion. The method has the advantages that according to the scheme, the whole system serves as an analysis object of data consanguinity, source tracing and tail tracing are truly achieved, the system automatically and rapidly analyzes changed data processing tasks again, the latest data consanguinity is input into a database, and timeliness of the data consanguinity is ensured.

Description

Technical Field

[0001] The present invention relates to the field of data lineage analysis, and in particular to a metadata lineage automatic generation tool based on grammatical rules. Background Art

[0002] Data lineage refers to the relationship between each data element and its source and destination during data processing. It describes the flow of data throughout the entire processing process. In layman's terms, data lineage is the hierarchical and traceable connection formed in the process of data generation, processing, and flow to final consumption.

[0003] Data lineage is an important basic capability within an organization to make data valuable. A mature data lineage system can help developers quickly locate problems, track data changes, determine upstream and downstream impacts, etc. The current mainstream data lineage technologies include automatic parsing, system tracking, machine learning methods, and manual collection. Automatic parsing is the main collection method at present, specifically parsing SQL statements, stored procedures, ETL (data extraction, transformation, and loading) processes and other files. Due to factors such as complex code and application environment, according to research, automatic parsing can cover 70-95% of data lineage, and it is currently impossible to achieve 100%. The system tracking method is that during the data processing flow, the processing subject tool is responsible for sending data mapping. The advantages are accurate, timely, and fine-grained collection, but not every tool can be integrated. The machine learning method calculates the similarity of data based on the dependency relationship between data sets. The advantage of this method is that it has no dependence on tools and businesses, but it requires manual confirmation of accuracy. Manual collection is the most primitive method of lineage analysis, which has the advantages of high accuracy and easy update and maintenance of rules, but its disadvantages are also obvious. It is not only time-consuming and labor-intensive, but also prone to errors. In addition, once the data processing logic changes, the lineage needs to be maintained again.

[0004] Data lineage describes the flow of data throughout the entire processing process. By building lineage, data can be better understood and managed, and it helps to improve data quality, ensure data security, strengthen data governance, etc. Therefore, it is very important for enterprises to establish and maintain data lineage. Based on this, the present invention designs a metadata lineage automatic generation tool based on grammatical rules. Summary of the invention

[0005] The purpose of the present invention is to solve the problems in the prior art and to propose a metadata lineage automatic generation tool based on grammatical rules.

[0006] A metadata lineage automatic generation tool based on grammatical rules includes the following steps:

[0007] S1: The data processing process is automatically collected through the interface, and the unique ID and version number of each processing task are collected at the same time. When the data processing logic changes, the system can respond in real time by monitoring the ID and version number of each task in real time, re-extract the changed data processing tasks and perform subsequent lineage analysis;

[0008] S2: Stores the automatically collected data processing tasks. At the same time, for each data processing task, stores the timestamp of its entry into the database and retains each version of the task data.

[0009] S3: Lineage analysis: Analyze the SQL statements of the processed data and convert them into instructions that the computer can understand and execute, help understand the structure and meaning of the SQL statements, and then deduce the lineage relationship between the data;

[0010] S4: Visualizing kinship data through kinship parsing criteria.

[0011] In the above-mentioned metadata lineage automatic generation tool based on grammatical rules, the visualized lineage data adopts a data lineage graph.

[0012] In the above-mentioned metadata lineage automatic generation tool based on grammatical rules, the SQL statement analysis includes:

[0013] SQL statement lexical analysis, combining the words and symbols obtained from lexical analysis into a syntax tree. The syntax tree is a tree structure with the SQL statement as the root node, and each child node represents a different part of the SQL statement;

[0014] SQL statement syntax analysis combines the words and symbols obtained from lexical analysis into a syntax tree according to the syntax rules. The syntax tree is a tree structure with the SQL statement as the root node, and each child node represents a different part of the SQL statement. Through syntax analysis, it can be determined whether the SQL statement conforms to the syntax rules and converted into the form of a syntax tree;

[0015] SQL statement semantic analysis performs semantic checks on SQL statements to determine whether there are semantic errors in the SQL statements. At the same time, it also parses the tables and columns in the SQL statements to determine their actual physical locations.

[0016] In the above-mentioned metadata lineage automatic generation tool based on grammatical rules, the lineage analysis criteria include table-level lineage analysis criteria and field-level lineage analysis criteria. The table-level lineage analysis criteria include:

[0017] Tables created in the script using the create with as statement will be identified as temporary tables and will not be directly related to the target table.

[0018] All formal tables in the script establish a blood relationship with the target table. The associated tables such as from and join must be identified, and temporary tables must be removed to establish blood relationships.

[0019] In the above-mentioned metadata lineage automatic generation tool based on grammatical rules, the field-level lineage parsing criteria include:

[0020] For the syntax of insert followed by SQL statement, the target table field-level lineage establishes a lineage relationship with the field of the source table directly fetched, and the field-level lineage resolution is not case sensitive to the English name of the field;

[0021] For the syntax of insert followed by enumeration value, there is no lineage mapping source table and field;

[0022] For the subquery filter conditions of keywords such as in and exists, the fields of the source table do not establish a blood relationship with the fields of the target table;

[0023] For the SQL statement syntax followed by update, establish the field relationship, extract the fields after set to establish the source table field and the target table field relationship;

[0024] For the lineage penetration syntax of the temporary table created by the create statement, the lineage inheritance principle of the temporary table fields is followed layer by layer;

[0025] For the temporary table lineage penetration syntax of the subquery spliced ​​after from, join, etc., the lineage relationship of the temporary table fields is inherited layer by layer according to the principle;

[0026] For the union subquery and set lineage parsing syntax, the lineage superposition principle of the source table fields in multiple subqueries is followed.

[0027] In the above-mentioned metadata lineage automatic generation tool based on grammatical rules, the data lineage graph includes:

[0028] Data nodes mark the specific information of data, such as owner, hierarchical information, and terminal information. Data nodes include master nodes, data inflow nodes, and data outflow nodes.

[0029] The flow line marks the flow path of data, which usually converges from the inflow node to the main node, and then spreads from the main node to the outflow node. In the flow line, not only the flow direction and flow relationship of the data can be marked, but also the data magnitude and update frequency can be marked by the thickness and length of the line;

[0030] Processing nodes mark the processing methods and rules during data flow. They are usually used for the flow lines between data nodes. Through processing nodes, you can intuitively understand what rules are used to process data when it flows between two nodes.

[0031] In the above-mentioned metadata lineage automatic generation tool based on grammatical rules, in the SQL statement lexical analysis, the SQL is split into four parts, including: temporary table processing logic part, SELECT statement part, FROM statement part, and common part, wherein the common part includes the filtering conditions of the entire table, and all fields are established with it. Then, the fields and table names in the above four parts of SQL are preliminarily parsed, and the data is stored in the container;

[0032] In SQL statement syntax and semantic analysis, SQL syntax analysis and semantic analysis are unified into one step, including:

[0033] S1: Format SQL statements to ensure that the SQL has a consistent structure before parsing, including removing comments and unnecessary whitespace;

[0034] S2: Use the sqlparse engineering package to parse SQL statements. After parsing, the SQL statements are parsed into a syntax tree. The parsing results are stored in the form of tuples, and the SQL statements are parsed into different tokens.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] (1) Comprehensiveness. The data processing process is actually the process of transferring, calculating, deducing and archiving data. The fluidity of data and the complex relationships between data will cause a slight change in a certain data to cause changes in data in multiple systems. In order to ensure the integrity of data lineage, this solution takes the entire system as the analysis object of data lineage, truly tracing the source and the end.

[0037] (2) Timeliness. The relationships between data and data may change at any time. In order to ensure the accuracy and availability of data lineage, lineage analysis must be updated synchronously with the data to ensure that the analysis results of data lineage are based on the latest data and data relationships. This solution monitors the data processing flow in real time. Once the data processing logic changes, the system will automatically and quickly re-analyze the changed data processing tasks and enter the latest data lineage relationships into the database to ensure the timeliness of data lineage.

[0038] (3) Applicability. There are many technologies and implementations for lineage analysis, and the breadth, depth, and dimensions of the analysis are also different, but all technologies serve the needs. Lineage analysis needs to be carried out on the premise of achieving the required goals. It can cover the needs of different scenarios, and the lineage analysis criteria listed in the solution are configurable. In the application, the most valuable data lineage can be obtained according to actual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a structural schematic diagram of a metadata lineage automatic generation tool based on grammatical rules proposed by the present invention.

[0040] Figure 2 This is a storage example diagram of the data ETL processing process in a metadata lineage automatic generation tool based on grammatical rules proposed by the present invention.

[0041] Figure 3 This is an example diagram of SQL field parsing in a metadata lineage automatic generation tool based on grammatical rules proposed by the present invention.

[0042] Figure 4 This is an example diagram of SQL table name parsing in a metadata lineage automatic generation tool based on grammatical rules proposed by the present invention.

[0043] Figure 5 This is an example diagram of SQL parsing results in a metadata lineage automatic generation tool based on grammatical rules proposed by the present invention.

[0044] Figure 6 This is an example diagram of blood relationship deduction in a metadata blood relationship automatic generation tool based on grammatical rules proposed by the present invention.

[0045] Figure 7 This is a schematic diagram of writing bloodline relationships into Oracle tables in a metadata bloodline automatic generation tool based on grammatical rules proposed by the present invention.

[0046] Figure 8 This is an example diagram of blood relationship storage in a metadata blood relationship automatic generation tool based on grammatical rules proposed by the present invention. DETAILED DESCRIPTION

[0047] Reference Figure 1-8 , a metadata lineage automatic generation tool based on grammatical rules, characterized by comprising the following steps:

[0048] S1: The data processing process is automatically collected through the interface, and the unique ID and version number of each processing task are collected at the same time. When the data processing logic changes, the system can respond in real time by monitoring the ID and version number of each task in real time, re-extract the changed data processing tasks and perform subsequent lineage analysis;

[0049] S2: Stores the automatically collected data processing tasks. At the same time, for each data processing task, stores the timestamp of its entry into the database and retains each version of the task data.

[0050] S3: Lineage analysis: Analyze the SQL statements of the processed data and convert them into instructions that the computer can understand and execute, help understand the structure and meaning of the SQL statements, and then deduce the lineage relationship between the data;

[0051] S4: Visualizing kinship data through kinship parsing criteria.

[0052] Among them, metadata lineage analysis starts with preparation work, which mainly involves collecting and storing the data processing process. First, the data processing process is automatically collected through the interface. Here, it is necessary to collect the unique ID and version number of each processing task at the same time. When the data processing logic changes, the system can respond in real time by monitoring the ID and version number of each task in real time, re-extract the changed data processing tasks, and perform subsequent lineage analysis. Secondly, the automatically collected data processing tasks are stored. For each data processing task, its entry timestamp should be stored. Retaining each version of the task data can help trace the source and locate historical problems. In addition, for large enterprises, there are tens of thousands of data processing tasks, so how to store this data is also a question worth pondering.

[0053] Then the data lineage analysis is performed, where data lineage analysis is essentially SQL analysis, which performs lexical analysis and grammatical analysis on the SQL statements that process the data, converting them into instructions that the computer can understand and execute, helping to understand the structure and meaning of the SQL statements, and then derive the lineage relationship between the data.

[0054] When in use, the blood relationship analysis rules adopted by this solution are aligned, and the specific analysis rules include:

[0055] Table-level blood relationship analysis criteria:

[0056] Tables created in the script using the create|with as statement will be identified as temporary tables and will not be directly related to the target table.

[0057] All formal tables in the script establish a blood relationship with the target table. The associated tables such as from and join must be identified, and temporary tables must be removed to establish the blood relationship.

[0058] Field-level lineage resolution criteria:

[0059] For the syntax of insert followed by SQL statement, the target table field-level lineage establishes a lineage relationship with the field of the source table directly fetched, and the field-level lineage resolution is not case sensitive to the English name of the field;

[0060] For the syntax of insert followed by enumeration value, there is no lineage mapping source table and field;

[0061] For the subquery filter conditions of keywords such as in and exists, the fields of the source table do not establish a blood relationship with the fields of the target table;

[0062] For the SQL statement syntax followed by update, establish the field relationship, extract the fields after set to establish the source table field and the target table field relationship;

[0063] For the lineage penetration syntax of the temporary table created by the create statement, the lineage inheritance principle of the temporary table fields is followed layer by layer;

[0064] For the temporary table lineage penetration syntax of the subquery spliced ​​after from, join, etc., the lineage relationship of the temporary table fields is inherited layer by layer according to the principle;

[0065] For the union subquery and set lineage parsing syntax, the lineage superposition principle of the source table fields in multiple subqueries is followed.

[0066] In the lexical analysis of SQL statements, the SQL statements are segmented according to the lexical rules, and the keywords, table names, column names, etc. are identified. Through lexical analysis, the SQL statements can be decomposed into individual words and symbols to prepare for the subsequent syntax analysis.

[0067] In SQL statement syntax analysis, the words and symbols obtained from lexical analysis are combined into a syntax tree according to the syntax rules. The syntax tree is a tree structure with the SQL statement as the root node and each child node representing the different parts of the SQL statement. Through syntax analysis, it is possible to determine whether the SQL statement conforms to the syntax rules and convert it into the form of a syntax tree.

[0068] In the semantic analysis of SQL statements, in the semantic analysis stage, the SQL statements are semantically checked to determine whether there are semantic errors in the SQL statements. At the same time, the tables and columns in the SQL statements are parsed to determine their actual physical locations.

[0069] After completing SQL parsing, the blood relationship between data can be deduced based on the syntax tree and semantic information, realizing blood relationship deduction and blood relationship storage. By analyzing the tables and columns in the SQL statement, the source and destination of the data can be determined, and then the transmission path between the data can be deduced. Finally, the obtained blood relationship is stored in the database according to a certain design, and the timestamp of entry is recorded.

[0070] When performing data lineage SQL parsing, you also need to complete the following steps. First, before performing SQL parsing, you need to ensure the correctness of the SQL statement. Second, process complex SQL statements. For complex SQL statements, you need to parse the SQL statements step by step, split them into simple statement fragments, and then deduce the lineage relationship. Finally, you need to consider the characteristics of the database. When performing data lineage SQL parsing, consider the characteristics of the database.

[0071] After the lineage analysis is completed, it is necessary to rely on visualization technology to clearly and intuitively convey the analysis results to users, helping them to conduct secondary analysis and specific applications. Data lineage maps are the most commonly used visualization solutions in lineage analysis. Differences in business needs will determine the differences in lineage analysis levels and lineage levels, which will be reflected in data lineage maps. Therefore, data lineage maps should also be layered based on data lineage levels, and intuitively present the lineage relationship of data from the application level, data level, and field level.

[0072] In specific applications, due to differences in business needs and the lineage information that can be collected and analyzed, the presentation of data lineage maps may vary, but their overall form is basically the same: with a certain data as the core node, it reflects the data source, data destination, flow path, and processing methods and processing in the path. Therefore, the data lineage visualization view should contain at least the following elements:

[0073] ① Data nodes. Data nodes mark specific information of data, such as owner, hierarchical information, terminal information, etc. The information of data nodes varies according to different lineage levels and business requirements. According to different data types, data nodes can be divided into master nodes, data inflow nodes, and data outflow nodes.

[0074] The master node is the core of the data lineage map. It is the data that the user currently needs to observe. There is only one master node, and the entire map presents its lineage relationship. The master node should be switchable and convenient. The data inflow node marks the source of the data of the master node. It is the parent node of the master node. It may have multiple or even multiple layers. The data outflow node marks the destination of the data of the master node. It is the child node of the master node. It may also have multiple or multiple layers. There is a special terminal node in the data outflow node. After the data reaches the terminal node, it will no longer flow elsewhere.

[0075] ② Flow line. The flow path of the marked data usually converges from the inflow node to the main node, and then spreads from the main node to the outflow node. In the flow line, not only the flow direction and flow relationship of the data can be marked, but also the data magnitude and update frequency can be marked by the thickness and length of the line.

[0076] ③Processing nodes. Mark the processing methods and processing rules during the data flow process, usually used for the flow lines between data nodes. Through the processing nodes, you can intuitively understand what rules are used to process the data when it flows between two nodes.

[0077] Specific application of blood relationship

[0078] ①Data traceability analysis

[0079] When data anomalies occur, we need to be able to track the cause of the anomaly and control the risk at an appropriate level. Relying on the plasticity of data lineage and according to the data link relationship in the lineage, the source and destination of the specified data can be traced, which can help users understand the meaning of the data, locate data problems in the whole process, and conduct data association impact analysis, etc., to solve the problem that data after multi-layer complex logic processing is difficult to understand, difficult to apply, and difficult to locate problems.

[0080] ②Data value assessment

[0081] Data value is the core standard of data management. Whether it is data pricing in data transactions or the protection level of data security, data value is an important reference factor. Therefore, how to accurately evaluate data value has become a major problem facing enterprises. Traditional data value assessment often relies entirely on relevant regulatory requirements and business experience, lacks assessment basis in specific application scenarios, and data value assessment is divorced from the application scenarios and real business value of the data. Data lineage provides a value assessment method based on the actual application of data: data with more users (demand side), greater usage, and more frequent updates are often more valuable.

[0082] Data audience: In the blood relationship diagram, the data outflow node on the right represents the audience, that is, the data demander. The more data demanders there are, the greater the data value.

[0083] Data update magnitude: In the data lineage diagram, the thicker the line of the data flow line, the greater the magnitude of the data update, which reflects the value of the data to a certain extent;

[0084] Data update frequency: The more frequently the data is updated, the fresher the data is and the higher its value is. On the blood relationship diagram, the shorter the line segment of the data flow line is, the more frequently it is updated.

[0085] ③Data quality assessment

[0086] Data lineage clearly records the data source as well as the processing methods and rules during data flow, and can realize the analysis of each data node and data quality assessment.

[0087] ④Data archiving reference

[0088] Data lineage records the whereabouts of data, which can clearly grasp the consumption of data. Once the data has no consumers, it means that the data has lost its value. At this point, the data can be further evaluated and considered for archiving or destruction.

[0089] In the implementation of the lineage analysis preparation work, the interface is used to automatically collect data ETL processing process and store it in the Oracle database table, mainly including ETL unique ID, version number, storage time, ETL processing logic, whether it is the latest version, etc. Figure 2 The version number of each latest version of the ETL task is monitored in real time. If the version number changes, the interface will be triggered to re-collect the changed ETL into the warehouse and update the latest version field.

[0090] In the implementation of blood relationship analysis, the specific implementation process of this solution uses Python as a technical means to perform SQL analysis, and uses the Python engineering package cx_Oracle to read ETL data and store it in the container.

[0091] The first step is to analyze the SQL statement. Split the SQL into four parts, including: temporary table processing logic part, SELECT statement part, FROM statement part, and common part (that is, the filter condition of the entire table, all fields are related to it). Perform preliminary analysis on the fields and table names in the above four parts of SQL, and store the data in a container, such as Figure 3 and Figure 4 As shown.

[0092] The second step is to analyze the syntax and semantics of SQL statements. In actual SQL parsing, SQL syntax analysis and semantic analysis can be combined into one step. First, the SQL statement is formatted to ensure that the SQL has a consistent structure before parsing, including removing comments and unnecessary spaces; then, the SQL statement is parsed using the sqlparse engineering package. After parsing, the SQL statement is parsed into a syntax tree. The parsing results are stored in the form of tuples. The SQL statement is parsed into different tokens, each of which has its own attributes (such as Keyword, Text, etc.). Figure 5 shown.

[0093] The third step is to deduce and store the blood relationship. Based on the SQL syntax tree, the mapping relationship between fields and tables and the association relationship between tables can be determined. The blood relationship analysis result is shown in the following example. Figure 6 Finally, the results of the blood relationship analysis are written into the Oracle table, as shown in Figure 7 and Figure 8As shown, the blood relationship storage table records the target table, target field, source table, source field and storage time information.

[0094] It is known from common technical knowledge that the present invention can be implemented by other embodiments that do not deviate from its spirit or essential features. Therefore, the above disclosed embodiments are only illustrative in all respects and are not exclusive. All changes within the scope of the present invention or within the scope equivalent to the present invention are included in the present invention.

Claims

1. A metadata lineage automatic generation tool based on grammatical rules, characterized in that: The following steps are involved: S1: The data processing process is automatically collected through the interface, and the unique ID and version number of each processing task are collected at the same time. When the data processing logic changes, the system can respond in real time by monitoring the ID and version number of each task in real time, re-extract the changed data processing tasks and perform subsequent lineage analysis; S2: Stores the automatically collected data processing tasks. At the same time, for each data processing task, stores the timestamp of its entry into the database and retains each version of the task data. S3: Lineage analysis: Analyze the SQL statements of the processed data and convert them into instructions that the computer can understand and execute, help understand the structure and meaning of the SQL statements, and then deduce the lineage relationship between the data; S4: Visualizing kinship data through kinship parsing criteria.

2. The tool for automatically generating metadata lineage based on grammatical rules according to claim 1, characterized in that: The visualized bloodline data adopts a data bloodline graph.

3. The tool for automatically generating metadata lineage based on grammatical rules according to claim 1, characterized in that: The SQL statement analysis includes: SQL statement lexical analysis, combining the words and symbols obtained from the lexical analysis into a syntax tree. The syntax tree is a tree structure with the SQL statement as the root node and each child node representing a different part of the SQL statement; SQL statement syntax analysis: According to the syntax rules, the words and symbols obtained from the lexical analysis are combined into a syntax tree. The syntax tree is a tree structure with the SQL statement as the root node, and each child node represents a different part of the SQL statement. Through syntax analysis, it can be determined whether the SQL statement conforms to the syntax rules and converted into the form of a syntax tree; SQL statement semantic analysis performs semantic checks on SQL statements to determine whether there are semantic errors in the SQL statements. At the same time, it also parses the tables and columns in the SQL statements to determine their actual physical locations.

4. The tool for automatically generating metadata lineage based on grammatical rules according to claim 1, characterized in that: The lineage analysis criteria include table-level lineage analysis criteria and field-level lineage analysis criteria. The table-level lineage analysis criteria include: a. Tables created by the create with as statement in the script will be identified as temporary tables and will not be directly related to the target table; b. All formal tables in the script establish a blood relationship with the target table. The associated tables including from and join must be identified, and the temporary tables must be eliminated to establish the blood relationship.

5. The tool for automatically generating metadata lineage based on grammatical rules according to claim 4, characterized in that: The field-level lineage resolution criteria include: a. For the SQL statement syntax followed by insert, the target table field-level lineage establishes a lineage relationship with the field of the source table directly fetched, and the field-level lineage resolution is not case sensitive to the English name of the field; b. For the syntax of insert followed by enumeration value, there is no lineage mapping source table and field; c. For subquery filter conditions that include the keywords "in" and "exists", the source table fields do not establish a blood relationship with the target table fields; d. For the SQL statement syntax followed by update, establish the field relationship, extract the fields after set, and establish the relationship between the source table field and the target table field; e. For the lineage penetration syntax of the temporary table created by the create statement, the lineage inheritance principle of the temporary table fields is followed; f. For the temporary table lineage penetration syntax of the subquery including from and join, the lineage inheritance principle of the temporary table fields is followed; g. For the union subquery and set lineage parsing syntax, follow the principle of superposition of source table field lineage relationships in multiple subqueries.

6. The tool for automatically generating metadata lineage based on grammatical rules according to claim 2, characterized in that: The data lineage graph includes: Data nodes mark the specific information of data, including owner, hierarchical information, and terminal information. Data nodes include master nodes, data inflow nodes, and data outflow nodes. The flow line marks the flow path of the data, which converges from the inflow node to the main node and then spreads from the main node to the outflow node; Processing nodes mark the processing methods and rules during data flow and are used for flow lines between data nodes.

7. The tool for automatically generating metadata lineage based on grammatical rules according to claim 3, characterized in that: In the SQL statement lexical analysis, the SQL is split into four parts, including: temporary table processing logic part, SELECT statement part, FROM statement part, and common part. The common part includes the filtering conditions of the entire table, and all fields are associated with it. Then, the fields and table names in the above four parts of SQL are preliminarily parsed, and the data is stored in the container; In SQL statement syntax and semantic analysis, SQL syntax analysis and semantic analysis are unified into one step, including: S1: Format SQL statements to ensure that the SQL has a consistent structure before parsing, including removing comments and unnecessary whitespace; S2: Use the sqlparse engineering package to parse SQL statements. After parsing, the SQL statements are parsed into a syntax tree. The parsing results are stored in the form of tuples, and the SQL statements are parsed into different tokens.

Images