Data blood relationship construction method and system based on NL2SQL and data blood relationship analysis method and system based on NL2SQL

Abstract

The invention relates to the technical field of data governance, in particular to an NL2SQL-based data blood relationship construction and analysis method and system, and the method comprises the following steps: natural language processing, NL2SQL conversion, SQL analysis, blood relationship storage and query, dynamic blood relationship updating, warning, auditing and safety, and blood relationship analysis and visual display. The method has the beneficial effects that the SQL statement is automatically generated through natural language query, and the blood relationship is analyzed, so that the tedious process of manual carding and static configuration is avoided, and the construction efficiency of the data blood relationship is greatly improved; the advanced natural language processing technology and the SQL analysis algorithm are utilized, the blood relationship between the data can be accurately extracted, the possibility of human errors is reduced, and the accuracy and reliability of the data blood relationship are improved.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and specifically to a data lineage construction and analysis method and system based on NL2SQL. Background Art

[0002] In today's information-based society, data has become a vital asset, widely used across various industries and fields. Data lineage plays a crucial role in data management and security. By displaying the source of data, the relationships between data, and how data changes at different stages of processing, it helps data managers understand the context of data and ensure its accuracy, integrity, and traceability.

[0003] Traditional bloodline analysis relies on manual parsing of SQL statements or database metadata, and has problems such as low efficiency, insufficient coverage, poor real-time performance, invisible dynamic data links, insufficient semantic understanding and ambiguity processing, traceability barriers for multi-source heterogeneous data, and lack of security audits and compliance. Summary of the Invention

[0004] The purpose of the present invention is to provide a data lineage construction and analysis method and system based on NL2SQL to solve the problems raised in the above background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a data lineage construction and analysis method based on NL2SQL, comprising the following steps:

[0006] Natural language processing: Converts user-entered natural language text into a semantic query graph that can be used for lineage analysis. This includes word segmentation and part-of-speech tagging, entity recognition, relationship extraction, semantic disambiguation, and semantic query graph generation.

[0007] NL2SQL conversion: Based on the preprocessing results of the natural language processing module, the natural language query is converted into SQL statements using the NL2SQL model. The NL2SQL model can adopt a rule-based, neural network-based, or pre-trained language model-based approach.

[0008] SQL parsing: Parsing the SQL statements entered by the user and extracting the data lineage information, specifically covering lexical analysis, syntax analysis, semantic analysis, and generating the SQL lineage graph;

[0009] Lineage storage and query: Use graph databases to store data lineage information and provide query interfaces, including data model design, data import, and query interface design;

[0010] Dynamic lineage update: Real-time monitoring of data operations and timely update of data lineage maps. Specific implementation includes data operation monitoring, lineage information extraction, and map updates. Incremental merging algorithms are used to incrementally write lineage data.

[0011] Alerting, auditing, and security: Provides alerting, auditing, and security management of data lineage information to ensure its security and compliance. Specific functions include alerting mechanisms, audit records, and security management.

[0012] Blood relationship analysis and visualization: Analyze the stored blood relationship information and use visualization tools to visualize the blood relationship map.

[0013] Preferably, the semantic disambiguation step of natural language processing specifically comprises: integrating a business knowledge base, wherein the business knowledge base includes a data dictionary, field annotations, and a business term list for semantic disambiguation and context understanding, and processing ambiguous problems in natural language;

[0014] The following disambiguation strategies are used to determine the exact meaning of each entity: Hierarchical priority: Determine the priority order of field-level lineage > table-level lineage > task-level lineage. When entities with the same name appear, judgment is made based on this priority order; Context association: Match the business domain to which the entity belongs based on the business line keywords in the query to clarify the meaning of the entity in a specific business environment; Confidence voting: Generate multiple possible mappings for entities with the same name, and determine the optimal solution through weighted voting based on the number of knowledge base references and field type matching to resolve the ambiguity problem of entities with the same name but different meanings.

[0015] Preferably, SQL parsing is responsible for parsing the SQL statements entered by the user and extracting data lineage information. The specific steps are as follows: Lexical analysis: decompose the SQL statement into lexical units, including keywords, table names, and field names; Syntax analysis: according to the SQL grammar rules, combine the lexical units into a syntax tree to represent the grammatical structure of the SQL statement; Semantic analysis: analyze the syntax tree to extract the association relationship between data tables, the source of the fields and the conversion rule data lineage information; generate an SQL lineage graph: convert the extracted lineage information into a graph structure, and merge it with the semantic query graph generated by the natural language processing module to form a complete data lineage graph foundation.

[0016] Preferably, dynamic lineage update uses an incremental merge algorithm to incrementally write lineage data, which specifically includes the following steps: conflict detection: compare the new lineage relationship with the existing map. When it is found that the field source is inconsistent, trigger the manual review process to ensure the accuracy of the lineage information; version management: use timestamp + operation type dual-dimensional version control to support backtracking of lineage relationships and facilitate the query of historical lineage relationships; batch write optimization: use Neo4j's UNWIND statement to batch import lineage relationships, improve the write performance to 50,000 records / second per node, and improve data update efficiency.

[0017] Preferably, alarm, audit and security have the following functions: Alarm mechanism: set alarm rules, and when the conditions of abnormal blood relationship and frequent data changes are met, send alarm notifications in time via email or SMS; Audit records: record all operations related to data blood relationship, including query, update, and deletion; track data change history, record modification time, modifier, and modification content information, and regularly generate audit reports for compliance inspection and problem tracing; Security management: control access to data blood relationship information, and limit access to sensitive information based on user roles and permissions; encrypt data blood relationship information for storage and transmission to ensure data security.

[0018] A system for constructing and analyzing data lineage based on NL2SQL, comprising:

[0019] Natural language processing module: used to convert user-entered natural language text into a semantic query graph that can be used for lineage analysis. This module includes word segmentation and part-of-speech tagging, entity recognition, relationship extraction, semantic disambiguation, and semantic query graph generation.

[0020] NL2SQL conversion module: Based on the preprocessing results of the natural language processing module, it uses the NL2SQL model to convert natural language queries into SQL statements. The NL2SQL model can adopt a rule-based, neural network-based, or pre-trained language model-based approach;

[0021] SQL parsing module: responsible for parsing the SQL statements entered by users and extracting data lineage information, specifically covering lexical analysis, syntax analysis, semantic analysis, and generating SQL lineage graph steps;

[0022] Lineage storage and query module: uses a graph database to store data lineage information and provides a query interface, including data model design, data import, and query interface design;

[0023] Dynamic lineage update module: monitors data operations in real time and updates the data lineage map in a timely manner. Specific implementations include data operation monitoring, lineage information extraction, and map updates. It also uses an incremental merge algorithm to incrementally write lineage data.

[0024] Alarm, Audit, and Security Module: This module is responsible for alarming, auditing, and security management of data lineage information to ensure the security and compliance of data lineage information. Specific functions include alarm mechanisms, audit records, and security management.

[0025] Blood relationship analysis and visualization display module: analyzes the stored blood relationship information and uses visualization tools to visualize the blood relationship map.

[0026] Preferably, the semantic disambiguation step of the natural language processing module specifically includes: integrating a business knowledge base, wherein the business knowledge base includes a data dictionary, field annotations, and a business term list for semantic disambiguation and context understanding, and processing ambiguous problems in natural language;

[0027] The following disambiguation strategies are used to determine the exact meaning of each entity: Hierarchical priority: Determine the priority order of field-level lineage > table-level lineage > task-level lineage. When entities with the same name appear, judgment is made based on this priority order; Context association: Match the business domain to which the entity belongs based on the business line keywords in the query to clarify the meaning of the entity in a specific business environment; Confidence voting: Generate multiple possible mappings for entities with the same name, and determine the optimal solution through weighted voting based on the number of knowledge base references and field type matching to resolve the ambiguity problem of entities with the same name but different meanings.

[0028] Preferably, the SQL parsing module is responsible for parsing the SQL statements input by the user and extracting data lineage information. The specific steps are as follows: Lexical analysis: decompose the SQL statement into lexical units, including keywords, table names, and field names; Syntax analysis: according to the SQL grammar rules, combine the lexical units into a syntax tree to represent the grammatical structure of the SQL statement; Semantic analysis: analyze the syntax tree to extract the association relationship between data tables, the source of the fields and the conversion rule data lineage information; generate an SQL lineage graph: convert the extracted lineage information into a graph structure, and merge it with the semantic query graph generated by the natural language processing module to form a complete data lineage graph foundation.

[0029] Preferably, the dynamic lineage update module uses an incremental merge algorithm to incrementally write lineage data, which specifically includes the following steps: conflict detection: comparing the new lineage relationship with the existing map. When it is found that the field source is inconsistent, the manual review process is triggered to ensure the accuracy of the lineage information; version management: using timestamp + operation type dual-dimensional version control, support backtracking of lineage relationships, and facilitate querying historical lineage relationships; batch write optimization: using Neo4j's UNWIND statement to batch import lineage relationships, the write performance is improved to 50,000 / second for a single node, thereby improving data update efficiency.

[0030] Preferably, the alarm, audit and security modules have the following functions: Alarm mechanism: Set alarm rules, and when the conditions of abnormal blood relationship and frequent data changes are met, send alarm notifications in time via email or SMS; Audit records: Record all operations related to data blood relationship, including query, update, and deletion; Track data change history, record modification time, modifier, and modification content information, and regularly generate audit reports for compliance inspection and problem tracing; Security management: Control access to data blood relationship information, and limit access to sensitive information based on user roles and permissions; Encrypt data blood relationship information for storage and transmission to ensure data security.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The NL2SQL-based data lineage construction and analysis method and system proposed in this paper automatically generates SQL statements and analyzes lineage relationships through natural language queries, avoiding the tedious process of manual combing and static configuration, greatly improving the efficiency of data lineage construction. Utilizing advanced natural language processing technology and SQL parsing algorithms, it can accurately extract lineage relationships between data, reducing the possibility of human error and improving the accuracy and reliability of data lineage relationships. It uses visualization tools to display lineage relationship maps, allowing users to intuitively understand the relationships and dependencies between data, facilitating data management and analysis. It provides multiple analysis functions such as impact analysis, traceability analysis, and quality analysis to help users better understand the value and risks of data and provide strong support for data decision-making. It integrates multiple data processing and analysis algorithms to efficiently process and analyze large-scale data and explore the potential value between data. Security measures such as identity authentication and authorization, data encryption, and security auditing and monitoring ensure the security and compliance of the system. At the same time, the audit tracking function can record user operation logs and data change history, providing guarantees for the compliance and security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0034] In order to clearly and completely describe the objectives and technical solutions of the present invention and make the advantages more clearly understood, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, not all of them, and are only used to explain the embodiments of the present invention, not to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] In the first embodiment, the present invention provides a technical solution: a data lineage construction and analysis method based on NL2SQL, comprising the following steps:

[0036] 1. Data source access and metadata collection

[0037] 1.1 Configure the NiFi Data Pipeline

[0038] Use the GetJDBC processor to connect to the MySQL database and synchronize the metadata of the "customer information table" regularly;

[0039] Use the FetchGit processor to pull the "Customer Data Synchronization Script.sql" from the ETL script repository.

[0040] 1.2 Metadata Cleaning and Storage

[0041] Parse SQL scripts to obtain the structural information of databases and tables in each data source, including table name, field name, field type, primary key, foreign key, etc., extract field mapping rules, generate field-level lineage, and store metadata in the MySQL metadata database.

[0042] Clean and transform the collected table structure information, remove redundant information and format errors, and convert it into a format supported by the graph database.

[0043] Node and edge creation: Based on the table structure information, nodes (tables, fields) and edges (dependencies, references) are created in the graph database. For example, for a foreign key relationship, an edge is created from the referencing table to the referenced table.

[0044] Data import: Use the import tool provided by the graph database or write an import script to import the processed data into the graph database.

[0045] 1.3 Collection and annotation of natural language query examples

[0046] Sample collection: Collect common natural language query examples within the enterprise, including query statements, update statements, delete statements, etc. This can be obtained through communication with business personnel and analysis of historical query logs.

[0047] Annotated data preparation: Each natural language query example is annotated with the corresponding SQL statement to ensure accuracy and completeness. This can be done manually or semi-automatically, using the existing NL2SQL model for preliminary annotation, followed by manual review and correction.

[0048] 2. Model training and optimization implementation

[0049] 2.1 Data Preprocessing

[0050] Text cleaning: Perform text cleaning on the collected natural language query examples and SQL statements to remove punctuation, special characters, stop words, etc., and only retain meaningful words.

[0051] Word segmentation and part-of-speech tagging: Use natural language processing tools (such as NLTK and SpaCy) to perform word segmentation and part-of-speech tagging on the cleaned text to provide features for subsequent model training.

[0052] 2.2 Model Selection and Training

[0053] Model selection: Choose an appropriate NL2SQL model based on project requirements and data characteristics, such as a pre-trained language model based on the Transformer architecture (BERT, T5).

[0054] Model training: Use the labeled data to train the selected model, setting appropriate training parameters such as learning rate, batch size, and number of training rounds. During the training process, monitor the model's loss function and accuracy, and adjust the training strategy in a timely manner.

[0055] 2.3 Model Evaluation and Optimization

[0056] Model evaluation: Use the test set to evaluate the trained model, calculate the model's accuracy, recall rate, F1 value and other indicators, and evaluate the model's performance.

[0057] Model optimization: Based on the evaluation results, the model is optimized. Model performance can be improved by adjusting the model structure, increasing training data, and refining the training algorithm. Additionally, the model is pruned and quantized to reduce the number of parameters and computational complexity, thereby increasing the model's inference speed.

[0058] 3. User query and blood relationship analysis implementation

[0059] 3.1 User Query Processing

[0060] Query reception: Users enter natural language queries through the web interface or API interface, and the system receives the query request.

[0061] Preprocessing: Preprocess the natural language queries entered by users, including word segmentation, part-of-speech tagging, named entity recognition, etc., to extract key entities and intents in the query.

[0062] NL2SQL conversion: Input the preprocessed query into the trained NL2SQL model to generate the corresponding SQL statement.

[0063] 3.2SQL Parsing and Blood Relationship Extraction

[0064] SQL parsing: Use SQL parsing tools (such as ANTLR, SQLParse) to convert the generated SQL statements into syntax trees.

[0065] Bloodline relationship extraction: Traverse the syntax tree, extract the data tables, fields, data flow and other information involved, and build a bloodline relationship map.

[0066] 3.3 Blood Relationship Analysis and Visualization

[0067] Bloodline relationship analysis: Analyzes stored bloodline relationship information, including impact analysis, traceability analysis, and quality analysis. For example, impact analysis can be used to determine which other data tables are affected by the modification of a data table; traceability analysis can be used to find the source of a certain data.

[0068] Visualization: Use visualization tools (such as D3.js and ECharts) to visualize the kinship map, allowing users to intuitively understand the relationships and dependencies between data. Interactive features such as zooming, dragging, and searching can be provided to facilitate user viewing and analysis of data.

[0069] 4. Dynamic lineage updates and alerts

[0070] Simulate data changes: Operations and maintenance personnel modify the ETL script; NiFi triggers stream processing through GitWebhook, parses the new script and generates new lineage rules.

[0071] Real-time monitoring and alerting: The dynamic lineage update module detects changes to field conversion rules, compares the old rules (AES) with the new rules (RSA), and triggers a "lineage rule change" alert. An alert notification is sent to the data team, including the changed field, old rules, new rules, and the scope of impact.

[0072] Bloodline graph update: Automatically update the bloodline edge attributes in the graph database, retain the old rule historical versions (distinguished by timestamps); use different colors in the visual graph to distinguish between new and old information.

[0073] 5. Alarm and audit implementation

[0074] 5.1 Alarm Mechanism Implementation

[0075] Alarm rule configuration: Configure rules for data quality alerts, data lineage change alerts, system performance alerts, and other alerts based on business needs. For example, you can set a threshold for the proportion of missing values ​​in a data table, and trigger an alert when the missing value ratio exceeds the threshold.

[0076] Alarm triggering and notification: Real-time monitoring of data quality, changes in data lineage relationships, and system performance indicators. When alarm rules are met, alarm notifications are triggered. Alarm information can be sent to relevant personnel via email, SMS, system messages, etc.

[0077] 5.2 Audit Trail Implementation

[0078] Operation log records: Record all user operations on the system, including query, update, delete, and other operations, as well as information such as the time, user, and content of the operation. This can be achieved using the database audit function or a custom logging module.

[0079] Data change tracking: Tracks data change history, recording information such as the time, person who modified the data, and the content of the modification. This can be achieved through database triggers or data version control tools.

[0080] Audit report generation: Regularly generate audit reports to summarize and analyze system operation logs and data changes. Audit reports can include user operation statistics, data change trends, security incident analysis, and other content to ensure system compliance and security.

[0081] 6. Security Implementation

[0082] 6.1 Identity Authentication and Authorization

[0083] Identity authentication: Use multi-factor authentication methods, such as username / password, SMS verification code, fingerprint recognition, etc. to ensure the authenticity of the user's identity. This can be achieved using third-party identity authentication services (such as OAuth and OpenIDConnect).

[0084] Authorization management: The role-based access control (RBAC) model manages user authorization, defines different roles (such as administrators and ordinary users) and permissions, and ensures that users can only access data and functions within their permission scope.

[0085] 6.2 Data Encryption

[0086] Data storage encryption: Sensitive data (such as user passwords and business data) is encrypted and stored using symmetric or asymmetric encryption algorithms such as AES and RSA.

[0087] Data transmission encryption: During data transmission, the HTTPS protocol is used to encrypt data to prevent data from being stolen or tampered with during transmission.

[0088] 6.3 Security Audit and Monitoring

[0089] Security event monitoring: Establish a security audit mechanism to monitor and record system security events in real time, such as failed logins and abnormal access. This can be achieved using a Security Information and Event Management (SIEM) system.

[0090] Security vulnerability scanning: Scan your system for security vulnerabilities regularly to identify and fix them promptly. You can use professional security scanning tools such as Nessus and OpenVAS.

[0091] Example 2, based on Example 1, proposes a system for constructing and analyzing data lineage based on NL2SQL, including:

[0092] a) Natural Language Processing Module: This module converts user-entered natural language text into a semantic query graph that can be used for lineage analysis. The specific steps are as follows:

[0093] Word segmentation and part-of-speech tagging: Use natural language processing tools (such as NLTK, HanLP, etc.) to segment the input natural language text and mark the part of speech of each word for subsequent entity recognition and relationship extraction.

[0094] Entity Recognition: Use named entity recognition (NER) technology to identify data entities in text, such as table names, field names, and data processing operations. You can use pre-trained NER models (such as BERT-NER) and optimize them in combination with the knowledge base of the business domain.

[0095] Relationship extraction: Analyze the semantic relationships between entities and determine the blood relationship of data, such as "originating from" and "influenced by". Relationship extraction can be performed using rule-based methods or machine learning models (such as SVM and LSTM).

[0096] Semantic disambiguation: The business knowledge base integrates a data dictionary, field annotations, and a business glossary for semantic disambiguation and contextual understanding. This helps address ambiguity in natural language, such as entities with the same name but different meanings. The business knowledge base and contextual information are used to determine the precise meaning of each entity. Disambiguation strategies primarily include the following:

[0097] Hierarchical priority: field-level lineage > table-level lineage > task-level lineage;

[0098] Contextual association: Match the business domain to which the entity belongs based on the business line keywords in the query (such as "marketing system" and "risk control model");

[0099] Confidence voting: Generate multiple possible mappings for entities with the same name, and determine the optimal solution through weighted voting based on the number of knowledge base references, field type matching, etc.

[0100] Generate a semantic query graph: Convert the identified entities and relationships into a graph structure, with each entity as a node and each relationship as an edge. The semantic query graph will serve as the basis for subsequent lineage analysis.

[0101] b) NL2SQL conversion:

[0102] Based on the preprocessing results, the NL2SQL model is used to convert natural language queries into SQL statements. The NL2SQL model can employ rule-based approaches, neural network-based approaches, or approaches based on pretrained language models. For example, a pretrained language model based on the Transformer architecture (such as BERT or T5) can be fine-tuned to understand natural language queries and generate corresponding SQL statements.

[0103] c) SQL parsing module:

[0104] This module is responsible for parsing the SQL statements entered by the user and extracting the data lineage information. The specific steps are as follows:

[0105] Lexical analysis: decomposes SQL statements into lexical units (tokens), such as keywords, table names, field names, etc.

[0106] Syntax analysis: According to the SQL grammar rules, lexical units are combined into a syntax tree (AST) to represent the grammatical structure of the SQL statement.

[0107] Semantic analysis: Analyze the syntax tree and extract data lineage information, such as the relationship between data tables, the source of fields, and conversion rules.

[0108] Generate SQL lineage graph: Convert the extracted lineage information into a graph structure and integrate it with the semantic query graph.

[0109] d) Bloodline storage and query module:

[0110] This module uses a graph database (such as Neo4j) to store data lineage information and provides a query interface. The specific design is as follows:

[0111] Data model design: Define the node and edge types in the graph database, such as data table nodes, field nodes, and blood relationship edges. Each node and edge can contain corresponding attributes, such as the name of the data table, the data type of the field, and the type of blood relationship, to accurately represent the blood relationship between data.

[0112] Data import: Import data from the semantic query graph and SQL lineage graph into the graph database to build a data lineage graph.

[0113] Query interface design: Provides multiple query interfaces, such as querying upstream and downstream data based on table name, querying lineage relationships based on field name, etc. Graph database queries can be performed using the Cypher query language.

[0114] e) Dynamic lineage update module:

[0115] This module monitors data operations in real time, such as database add, delete, modify, and query operations, and the execution of ETL tasks, and updates the data lineage map in a timely manner. The specific implementation is as follows:

[0116] Data operation monitoring: Capture data operation information through database log files (such as MySQL Binlog and PostgreSQL WAL log) or the monitoring interface of the ETL tool.

[0117] Bloodline information extraction: Extract bloodline-related information from monitored data operation information, such as changes in data tables, field updates, etc.

[0118] Graph update: Based on the extracted blood relationship information, the blood relationship graph in the graph database is updated to ensure the timeliness of the blood relationship information.

[0119] The incremental merge algorithm is used to incrementally write lineage data, as follows:

[0120] Conflict detection: Compare the new blood relationship with the existing map. If there is any inconsistency in the field source, the manual review process is triggered;

[0121] Version management: adopts timestamp + operation type dual-dimensional version control, supporting blood relationship backtracking;

[0122] Batch write optimization: Use Neo4j's UNWIND statement to batch import lineage relationships, improving write performance to 50,000 records per second per node.

[0123] f) Alarm, audit and security module:

[0124] This module is responsible for alarming, auditing, and security management of data lineage information to ensure the security and compliance of data lineage information. Specific functions are as follows:

[0125] Alarm mechanism: Set alarm rules, such as abnormal blood relationship, frequent data changes, etc. When the alarm conditions are met, timely send alarm notifications, such as emails, text messages, etc.

[0126] Audit records: Record all operations related to data lineage, such as queries, updates, and deletions. They also track data change history, recording information such as modification time, modification person, and modification content. Audit records can be used for post-event compliance checks and problem tracing. Regularly generated audit reports summarize and analyze system operation logs and data changes to ensure system compliance and security.

[0127] Security control: Access control is implemented on data lineage information, limiting access to sensitive data lineage information based on user roles and permissions. At the same time, data lineage information is encrypted for storage and transmission to ensure data security.

[0128] g) Blood relationship analysis and visualization:

[0129] Analyze the stored blood relationship information, including impact analysis, traceability analysis, quality analysis, etc. Use visualization tools (such as D3.js, ECharts, etc.) to visualize the blood relationship map, allowing users to intuitively understand the relationships and dependencies between data.

[0130] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A data lineage construction and analysis method based on NL2SQL, characterized by: The following steps are involved: Natural language processing: Converts user-entered natural language text into a semantic query graph that can be used for lineage analysis. This includes word segmentation and part-of-speech tagging, entity recognition, relationship extraction, semantic disambiguation, and semantic query graph generation. NL2SQL conversion: Based on the preprocessing results of the natural language processing module, the natural language query is converted into SQL statements using the NL2SQL model. The NL2SQL model can adopt a rule-based, neural network-based, or pre-trained language model-based approach. SQL parsing: Parsing the SQL statements entered by the user and extracting the data lineage information, specifically covering lexical analysis, syntax analysis, semantic analysis, and generating the SQL lineage graph; Lineage storage and query: Use graph databases to store data lineage information and provide query interfaces, including data model design, data import, and query interface design; Dynamic lineage update: Real-time monitoring of data operations and timely update of data lineage maps. Specific implementation includes data operation monitoring, lineage information extraction, and map updates. Incremental merging algorithms are used to incrementally write lineage data. Alerting, auditing, and security: Provides alerting, auditing, and security management of data lineage information to ensure its security and compliance. Specific functions include alerting mechanisms, audit records, and security management. Blood relationship analysis and visualization: Analyze the stored blood relationship information and use visualization tools to visualize the blood relationship map.

2. The data lineage construction and analysis method based on NL2SQL according to claim 1, characterized in that: The semantic disambiguation step of natural language processing specifically includes: integrating a business knowledge base, which includes a data dictionary, field annotations, and a business term list for semantic disambiguation and context understanding to handle ambiguity in natural language; The following disambiguation strategies are used to determine the exact meaning of each entity: Hierarchical priority: Determine the priority order of field-level lineage > table-level lineage > task-level lineage. When entities with the same name appear, judgment is made based on this priority order; Context association: Match the business domain to which the entity belongs based on the business line keywords in the query to clarify the meaning of the entity in a specific business environment; Confidence voting: Generate multiple possible mappings for entities with the same name, and determine the optimal solution through weighted voting based on the number of knowledge base references and field type matching to resolve the ambiguity problem of entities with the same name but different meanings.

3. The data lineage construction and analysis method based on NL2SQL according to claim 2, characterized in that: SQL parsing is responsible for parsing the SQL statements entered by the user and extracting data lineage information. The specific steps are as follows: Lexical analysis: decomposes the SQL statement into lexical units, including keywords, table names, and field names; Syntax analysis: according to SQL grammar rules, combines lexical units into a syntax tree to represent the grammatical structure of the SQL statement; Semantic analysis: analyzes the syntax tree to extract the relationship between data tables, the source of fields, and the conversion rules of data lineage information; Generate SQL lineage graph: Convert the extracted lineage information into a graph structure and integrate it with the semantic query graph generated by the natural language processing module to form a complete data lineage graph foundation.

4. The data lineage construction and analysis method based on NL2SQL according to claim 3, characterized in that: Dynamic lineage updates use an incremental merge algorithm to incrementally write lineage data, which specifically includes the following steps: Conflict detection: Compare the new lineage relationship with the existing map. When inconsistent field sources are found, a manual review process is triggered to ensure the accuracy of the lineage information; Version management: Use timestamp + operation type dual-dimensional version control to support backtracking of lineage relationships and facilitate the query of historical lineage relationships; Batch write optimization: Use Neo4j's UNWIND statement to batch import lineage relationships, improving write performance to 50,000 records / second per node, thereby improving data update efficiency.

5. The data lineage construction and analysis method based on NL2SQL according to claim 4, characterized in that: Alarm, audit, and security features include: Alarm mechanism: Set alarm rules to send timely alarm notifications via email or SMS when abnormal blood relationship or frequent data changes are found; Audit records: Record all operations related to data blood relationship, including query, update, and deletion; Track data change history, record modification time, modifier, and modification content, and regularly generate audit reports for compliance checks and problem tracing; Safety Management and control: Control access to data lineage information and restrict access to sensitive information based on user roles and permissions; Data lineage information is encrypted for storage and transmission to ensure data security.

6. A system for constructing and analyzing data lineage based on NL2SQL according to claim 5, characterized in that: include: Natural language processing module: used to convert user-entered natural language text into a semantic query graph that can be used for lineage analysis. This module includes word segmentation and part-of-speech tagging, entity recognition, relationship extraction, semantic disambiguation, and semantic query graph generation. NL2SQL conversion module: Based on the preprocessing results of the natural language processing module, it uses the NL2SQL model to convert natural language queries into SQL statements. The NL2SQL model can adopt a rule-based, neural network-based, or pre-trained language model-based approach; SQL parsing module: responsible for parsing the SQL statements entered by users and extracting data lineage information, specifically covering lexical analysis, syntax analysis, semantic analysis, and generating SQL lineage graph steps; Lineage storage and query module: uses a graph database to store data lineage information and provides a query interface, including data model design, data import, and query interface design; Dynamic lineage update module: monitors data operations in real time and updates the data lineage map in a timely manner. Specific implementations include data operation monitoring, lineage information extraction, and map updates. It also uses an incremental merge algorithm to incrementally write lineage data. Alarm, Audit, and Security Module: This module is responsible for alarming, auditing, and security management of data lineage information to ensure the security and compliance of data lineage information. Specific functions include alarm mechanisms, audit records, and security management. Blood relationship analysis and visualization display module: analyzes the stored blood relationship information and uses visualization tools to visualize the blood relationship map.

7. A system according to claim 6, characterized in that: The semantic disambiguation step of the natural language processing module is specifically as follows: integrating a business knowledge base, which includes a data dictionary, field annotations, and a business term list for semantic disambiguation and context understanding to handle ambiguity in natural language; The following disambiguation strategies are used to determine the exact meaning of each entity: Hierarchical priority: Determine the priority order of field-level lineage > table-level lineage > task-level lineage. When entities with the same name appear, judgment is made based on this priority order; Context association: Match the business domain to which the entity belongs based on the business line keywords in the query to clarify the meaning of the entity in a specific business environment; Confidence voting: Generate multiple possible mappings for entities with the same name, and determine the optimal solution through weighted voting based on the number of knowledge base references and field type matching to resolve the ambiguity problem of entities with the same name but different meanings.

8. A system according to claim 7, characterized in that: The SQL parsing module is responsible for parsing the SQL statements entered by the user and extracting data lineage information. The specific steps are as follows: Lexical analysis: decomposes the SQL statement into lexical units, including keywords, table names, and field names; Syntax analysis: according to the SQL grammar rules, combines lexical units into a syntax tree to represent the grammatical structure of the SQL statement; Semantic analysis: analyzes the syntax tree to extract the relationship between data tables, the source of the fields, and the conversion rules of the data lineage information; Generate SQL lineage graph: Convert the extracted lineage information into a graph structure and integrate it with the semantic query graph generated by the natural language processing module to form a complete data lineage graph foundation.

9. A system according to claim 8, characterized in that: The dynamic lineage update module uses an incremental merge algorithm to incrementally write lineage data, which specifically includes the following steps: Conflict detection: Compare the new lineage relationship with the existing map. When inconsistent field sources are found, a manual review process is triggered to ensure the accuracy of the lineage information; Version management: Use timestamp + operation type dual-dimensional version control to support backtracking of lineage relationships and facilitate the query of historical lineage relationships; Batch write optimization: Use Neo4j's UNWIND statement to batch import lineage relationships, improve write performance to 50,000 records / second per node, and improve data update efficiency.

10. A system according to claim 9, characterized in that: The alarm, audit, and security modules have the following functions: Alarm mechanism: Set alarm rules to send timely alarm notifications via email or SMS when abnormal blood relationship or frequent data changes are met; Audit records: Record all operations related to data blood relationship, including query, update, and deletion; Track data change history, record modification time, modifier, and modification content, and regularly generate audit reports for compliance checks and problem tracing; Safety Management and control: Control access to data lineage information and restrict access to sensitive information based on user roles and permissions; Data lineage information is encrypted for storage and transmission to ensure data security.

Images