A dynamic RAG-NL2SQL system based on real-time data pedigree

Abstract

This invention belongs to the field of computer data processing technology, specifically relating to a dynamic RAG-NL2SQL system based on a real-time data lineage graph. It includes: a cost-latency-aware intelligent routing module that receives user questions, performs macro-level meta-decision, and calls relevant modules for execution; a real-time data lineage graph construction and maintenance module that constructs and dynamically maintains the real-time data lineage graph; an intent-lineage dynamic retrieval enhancement module that queries the data source subgraph matching the user question from the real-time data lineage graph and serializes it into a text description; a traceable SQL generation and closed-loop verification module that generates SQL based on the text description and the user's original question, executes it, and verifies the execution result; if verification fails, a closed-loop verification and repair process is initiated; a zero-shot memory reuse module that constructs memory units from verified queries and dynamically evaluates these memory units; and performs semantic reuse and intelligent elimination based on the evaluation results. This invention improves the system's automation level and overall reliability.

Description

Technical Field

[0001] This invention belongs to the field of computer data processing technology, specifically relating to a dynamic RAG-NL2SQL system based on real-time data lineage graphs. Background Technology

[0002] Currently, NL2SQL technology, which utilizes natural language to interact with databases, is key to lowering the barrier to data access. However, existing technologies have the following significant drawbacks when dealing with complex and dynamic enterprise-level data environments:

[0003] 1. Blood relationship analysis remains at a static, descriptive level and lacks real-time guidance capabilities.

[0004] Technologies such as those that analyze historical logs (e.g., Chinese patent CN119166740A) advance by identifying "table lineage" through analyzing historical query statements (such as JOIN operations) and design documents to build a richer asset knowledge base. However, its core drawback is that this lineage is a historical, static snapshot. It can describe "a historical data flow relationship between table A and table B," but it cannot perceive the current health of this flow process in real time. For example, whether the ETL job executing this flow is delayed, has failed, or how fresh the output data is. Therefore, when a large model relies on this static knowledge base, it may still select a data link that is "interrupted" or "congested," leading to query failures or the acquisition of outdated data.

[0005] 2. The lack of verification of the generated results and a self-healing closed loop results in poor system reliability.

[0006] All existing NL2SQL systems, including those integrating static lineage knowledge, essentially operate on a one-way "translation" process: natural language input → SQL output → database execution. When the generated SQL returns empty or abnormal results due to upstream data issues (such as ETL job failures), the system itself cannot correlate the query failure with a specific data link failure. The entire system lacks an automated closed-loop mechanism for perception, location, and repair, leading to heavy reliance on manual troubleshooting and poor overall system reliability and user experience.

[0007] 3. Inquiry costs are uncontrollable, and resource utilization efficiency is low.

[0008] Current technologies typically focus only on the correct semantic translation when generating queries, neglecting the economic efficiency of query execution. A single natural language query can be transformed into a query that places a huge load on the database.

[0009] In summary, there is an urgent need for a RAG-NL2SQL system to address the three core challenges that existing technologies have failed to effectively solve in real-world business scenarios: real-time data link awareness, closed-loop verification of query results, and dynamic optimization of query costs. Summary of the Invention

[0010] To address the shortcomings of existing technologies, this invention proposes a dynamic RAG-NL2SQL system based on real-time data lineage graphs. This system includes:

[0011] A dynamic RAG-NL2SQL system based on real-time data lineage graphs, characterized in that it includes:

[0012] The module includes a real-time data lineage graph construction and maintenance module, an intent-lineage dynamic retrieval enhancement module, a traceable SQL generation and closed-loop verification module, a zero-sample memory reuse module, and a cost-delay-aware intelligent routing module.

[0013] The cost-latency-aware intelligent routing module receives user questions and performs macro-level meta-decision to obtain the optimal path to answer user questions and calls relevant modules to execute them.

[0014] The real-time data lineage graph construction and maintenance module is used to construct and dynamically maintain the real-time data lineage graph;

[0015] The intent-line dynamic retrieval enhancement module is used to query the data source subgraph that matches the user's question from the real-time data lineage graph and serialize the data source subgraph into a text description;

[0016] The traceable SQL generation and closed-loop verification module uses a large model to generate SQL based on text description and user's original question, executes the SQL and verifies the execution result. If the verification fails, the closed-loop verification and repair process is initiated.

[0017] The zero-sample memory reuse module is used to construct memory units from queries that have been verified by the traceable SQL generation and closed-loop verification module, and to dynamically evaluate the memory units; based on the evaluation results, semantic reuse and intelligent elimination are performed.

[0018] Preferably, the process by which the cost-latency-aware intelligent routing module receives user questions and makes macro-level meta-decision decisions includes:

[0019] The cost-latency-aware intelligent routing module calculates the cost of three paths in real time, respectively. Database execution cost of the path API call cost of the path and The cost of cached lookup for a path; The path is to retrieve the answer from the zero-sample memory; The path involves generating SQL via LLM and executing it in an internal database to retrieve the answer. The approach involves obtaining the answer through direct reasoning using LLM;

[0020] The second-level MAB decision method calculates the reward scores for the three paths based on their costs, and selects the path with the highest reward score as the best path to answer the user's question.

[0021] Furthermore, the formula for calculating the reward score is as follows:

[0022]

[0023] in, Indicates the reward score. Indicates cost weight. Indicates the delay weight. Represents the cost of the path. Indicates the path delay.

[0024] Preferably, the process of constructing and maintaining the real-time data lineage graph includes: taking various entities in the data ecosystem as nodes and the dependencies and flow relationships between various entities as edges; acquiring events in real time and calculating the health weight of each edge based on the event information to obtain the real-time data lineage graph.

[0025] Furthermore, the formula for calculating the health weight of an edge is:

[0026]

[0027] in, Represents the health weight of the edge. Represents the freshness factor. This represents the delay attenuation coefficient. The link quality score represents the edge. This indicates a delay in data freshness.

[0028] Preferably, the process by which the intent-line dynamic retrieval enhancement module queries a data source subgraph that matches the user's question from a real-time data lineage graph includes:

[0029] The user's question is embedded to obtain a question embedding vector; a vector index is built based on the real-time data lineage graph.

[0030] Based on the question embedding vector, an approximate nearest neighbor search is performed in the vector index of the real-time data lineage graph to obtain multiple candidate nodes;

[0031] Obtain the health weight of the upstream link of each candidate node, and select the candidate node corresponding to the link with the largest health weight as the final candidate node;

[0032] Starting from the final candidate node, perform a weighted graph traversal on the lineage graph to obtain the data source subgraph.

[0033] Preferably, the verification methods for the results of SQL execution by the traceable SQL generation and closed-loop verification module include: empty result verification, hash verification, or statistical verification; empty result verification refers to determining whether the result set of the executed SQL is empty, and if it is empty, the verification fails; hash verification refers to calculating the hash value of the result of the executed SQL and determining whether it is correct, and if it is incorrect, the verification fails; statistical verification refers to checking the consistency of the number of rows and the content of key numerical columns in the result of the executed SQL, and if they are inconsistent, the verification fails.

[0034] Preferably, the process of initiating the closed-loop verification and repair process by the traceable SQL generation and closed-loop verification module includes:

[0035] When verification fails, initiate closed-loop verification, parse the TLC attached to the SQL, and check the health status of each node in reverse order along the lineage starting from the final table to locate the faulty node.

[0036] Automatically trigger a one-click rollback or node-level recalculation operation for the faulty node to complete the repair;

[0037] Re-execute the SQL and verify it to return the answer to the question.

[0038] Preferably, the zero-sample memory reuse module dynamically evaluates memory units, and the process of semantic reuse and intelligent elimination based on the evaluation results includes:

[0039] The user's question is embedded to obtain a question embedding vector;

[0040] Search the memory database of the zero-sample memory reuse module to find the k non-expired memory units that are most similar to the problem vector and use them as k cache arms;

[0041] The complete NL2SQL link is used as the baseline arm. The first-level MAB decision selects the arm with the highest UCB value among k+1 arms as the execution result. After execution, the reward of the arm is updated according to the actual cost and delay; the UCB value of the arm is updated according to the reward.

[0042] If the UCB value of a cache arm is less than the preset threshold N times consecutively, the cache arm is removed from the memory database; otherwise, it is determined whether the hash value of the cache arm has changed after this execution. If it has not changed, the memory unit is retained; if it has changed, the hash value and timestamp of the memory unit are updated.

[0043] Preferably, the system also includes an exception triggering function, the implementation steps of which include:

[0044] The real-time data lineage graph construction and maintenance module monitors each data link in the real-time data lineage graph. When an anomaly is detected in the quality attributes of a data link, an analysis intent is generated.

[0045] The intent-based dynamic retrieval enhancement module retrieves subgraphs related to abnormal links from the real-time data lineage graph based on the analysis intent.

[0046] The traceable SQL generation and closed-loop verification module generates comparative analysis SQL based on the subgraph and analysis intent; the SQL is executed to obtain the analysis results.

[0047] The system integrates all information into structured alarm messages and sends them to the user.

[0048] The beneficial effects of this invention are as follows:

[0049] 1. Fundamental improvement in accuracy: By querying real-time, weighted lineage graphs instead of static metadata, the system can accurately locate the latest and most reliable data sources, upgrading the knowledge source of LLM from an "offline map" to a "real-time GPS". This fundamentally avoids the "table hallucination" problem caused by information lag and significantly reduces the query failure rate caused by selecting incorrect or outdated data sources.

[0050] 2. A Qualitative Leap in Reliability and Automation: The "SQL-lineage closed-loop verification" mechanism transforms the traditional "open translation" of NL2SQL into a "closed-loop self-healing system." It converts the manual troubleshooting that would take hours or even longer in the traditional model into automatic location and recovery in seconds, achieving "zero manual intervention" at the operation and maintenance level in most scenarios, greatly improving the system's automation level and overall reliability.

[0051] 3. Order-of-magnitude optimization of query efficiency: The zero-sample memory reuse module reduces the response time for high-frequency or semantically repetitive queries from seconds to sub-seconds, achieving an order-of-magnitude efficiency improvement. Simultaneously, the intelligent routing module further optimizes query paths, significantly reducing average query latency.

[0052] 4. Effective Control and Optimization of IT Costs: The cost-latency-aware routing module acts as an "intelligent economic scheduler," proactively avoiding high-load database queries and expensive API calls, thus preventing unnecessary resource consumption. While ensuring service quality, it transforms IT resources from "passive consumption" to "proactive optimization," significantly improving overall resource utilization efficiency. Attached Figure Description

[0053] Figure 1 This is the overall flowchart of the dynamic RAG-NL2SQL system based on real-time data lineage graph in this invention;

[0054] Figure 2 This is a timing diagram of the two-layer MAB decision in this invention;

[0055] Figure 3 This is a schematic diagram of the zero-sample memory reuse module in this invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] To overcome the fundamental deficiency of existing technologies where Large Language Models (LLMs) are disconnected from dynamic data ecosystems, this invention proposes a dynamic RAG-NL2SQL system based on real-time data lineage graphs. This invention aims to build a new generation of intelligent data analysis engine with deep data state awareness, self-verification of execution results, and dynamic optimization of operating costs, specifically achieving the following progressive goals:

[0058] 1. Laying the foundation for "perception": By constructing and maintaining a real-time data lineage graph that is updated in seconds, the dynamic health status of the data link (such as freshness and quality) is quantified into computable weights, providing a real-time and reliable knowledge source for large models, and fundamentally solving the "table illusion" problem caused by information lag.

[0059] 2. Establish an accountability loop: Building upon perception capabilities, establish a traceable SQL generation and closed-loop verification mechanism. Each generated SQL statement will be accompanied by its "Data Lineage Chain (TLC)." When query results are abnormal, this chain can automatically locate the upstream fault source and trigger a self-healing process, achieving a qualitative shift from "doing our best" to "being responsible for the results."

[0060] 3. Achieve "Optimization" Intelligence: Based on reliability, introduce zero-sample memory reuse and cost-latency-aware intelligent routing strategies to perform a global trade-off between economy and efficiency for query tasks. Dynamically select the optimal execution path based on query cost and real-time system load, ultimately maximizing both system resource utilization and user satisfaction.

[0061] like Figure 1As shown, the system's real-time data lineage graph construction and maintenance module, intent-lineage dynamic retrieval enhancement module, traceable SQL generation and closed-loop verification module, zero-sample memory reuse module, and cost-delay-aware intelligent routing module work together to form a complete, closed-loop, and self-optimizing processing flow. The following is a detailed description of each module:

[0062] (1) Cost-delay-aware intelligent routing module

[0063] This module is the system's "intelligent traffic command center." Before actually executing any data query task, it quickly evaluates all feasible paths and selects the current optimal route. It incorporates economic and operations research considerations into every decision of the system, aiming to achieve the best balance between cost, speed, and system load.

[0064] Upon receiving a user's question, this module makes a macro-level meta-decision: "What is the wisest way to answer this question?"; it then determines the optimal path to answer the user's question and calls the relevant modules to execute it.

[0065] Optional paths include:

[0066] Path A ( ): Retrieve the answer from a zero-shot memory optimized using the MAB strategy;

[0067] Path B ( ): Generate SQL using LLM and execute it in an internal database to obtain the answer;

[0068] Path C ( ): Obtain answers through direct reasoning using LLM, used to handle open-ended, summary questions that cannot be solved using SQL.

[0069] Specifically:

[0070] The cost-latency-aware intelligent routing module calculates the cost of three paths in real time, respectively. Database execution cost of the path API call cost of the path and The cost of cached lookup for the path.

[0071] Database execution cost: After the SQL is generated (but before execution), its execution plan is analyzed using the database's EXPLAIN command. Key information to focus on includes the expected number of rows scanned, whether indexes are used, the join type (e.g., Nested Loop, Hash Join), and whether temporary tables are generated. This information is then converted into a quantifiable cost score using a weighted model, representing the expected consumption of database I / O and CPU resources by the SQL. This is a predicted cost.

[0072] API call cost: This is not just a price list from API providers. A more accurate cost estimate is based on the number of tokens in the input question and the expected length of the output tokens. LLM API call cost The calculation formula is:

[0073]

[0074] in, This indicates the number of input tokens. This represents the price of each input token. This indicates the expected number of tokens to be output. This represents the price of each output token.

[0075] Cache query cost: The initial action of the path is to perform an efficient vector nearest neighbor search, which has extremely low computational and I / O costs. When making prediction decisions, it can be regarded as a startup cost close to 0.

[0076] Obtain the real-time status of the system:

[0077] Database real-time load ( ): Obtain the current number of active connections, query queue length, CPU utilization, etc. through monitoring tools (such as Prometheus) or database built-in views (such as pg_stat_activity).

[0078] API real-time latency ( ): Maintain a sliding average latency of recent API calls to reflect the current health of the API service and network quality.

[0079] The "brain" of the cost-latency-aware intelligent routing module is responsible for integrating all the above information and making the final decision. In a preferred embodiment of the invention, its design essence lies in constructing a two-tier multi-armed bandit (MAB) decision-making system, deeply integrating macro-level routing with micro-level cache management. The two-tier MAB decision-making timing is as follows: Figure 2As shown. The first layer of MAB decision-making is the "micro-caching sentinel," which operates within the zero-sample memory reuse module. Its core responsibility is selection and elimination. It dynamically evaluates each specific cache unit and compares it with the complete database execution path (baseline arm), automatically eliminating those "rigid" caches that have lost their efficiency advantage, thus ensuring... The path itself is of high quality and high efficiency. The second-layer MAB decision is "macro-level intelligent routing," which works in this module. After the first-layer MAB ensures the health of the cache pool, the MAB in this module takes the lead as the commander-in-chief, solving a more macro-level strategic choice problem.

[0080] The second-level MAB decision method calculates reward scores for the three paths based on their costs, and selects the path with the highest reward score as the optimal path to answer the user's question. The formula for calculating the reward score is:

[0081]

[0082] in, Indicates the reward score. Indicates cost weight. Indicates the delay weight. This represents the cost of the path (the sum of the costs of the three paths). Indicates the path latency (calculated from the database real-time load and API real-time latency).

[0083] Working principle analogy:

[0084] Imagine three macro-level "slot machines" (CACHE, DB, API), each with a different payout rate (i.e., "reward") that changes over time. The goal is to maximize the total reward within a finite number of attempts.

[0085] "Arms": Three macro-routing options , , .

[0086] "Action": Routes a query request to the corresponding path.

[0087] "Reward": After the query is executed, a reward score is calculated based on the actual cost and latency. The reward function is consistent with the first-level MAB to ensure the consistency of the evaluation system.

[0088] In this module, and It measures a certain macro-path ( , , The total end-to-end cost from start to finish. This is conceptually different from the micro-cost measured in the zero-shot memory reuse module, which aims to make strategic choices between different system-level paths.

[0089] The advantage of this module is that it elegantly solves the "exploitation-exploitation" dilemma.

[0090] This is because the model tends to select the macroscopic path with the highest historical average reward. For example, if... If it consistently performs well (low cost, fast speed), the model will frequently select it.

[0091] Exploration: The model will try paths with a relatively small probability that have historically performed poorly (e.g., This continuous "exploration" allows the system to sense changes in the environment (such as API provider price reductions or database load spikes) and dynamically adjust its optimal strategy.

[0092] The effects produced by this module include:

[0093] Layered intelligence with clear responsibilities: The two-layer MAB architecture is one of the core innovations of this invention. The first layer of MAB focuses on the micro-quality (tactical level) of cached assets, while the second layer of MAB focuses on the strategic selection of macro-paths. This layered design makes the system logic clear, the intelligent decision-making targeted, and the system easy to maintain and extend.

[0094] Dynamic adaptation: Static rules are "dead," while reinforcement learning models are "alive." They can dynamically adjust routing strategies based on real feedback (rewards) in each round, without human intervention.

[0095] Continuous self-optimization: It can automatically balance the contradiction between "utilizing" the known optimal path and "exploring" potential better paths, making the routing strategy of the entire system smarter over time.

[0096] Lightweight and real-time: The computational overhead of two-layer MAB models (such as UCB and Thompson Sampling) is extremely small, and the decision-making speed is within milliseconds, which fully meets the requirements of real-time decision-making before querying and will not introduce new performance bottlenecks.

[0097] By introducing this intelligent decision-making system with hierarchical self-learning capabilities, the system no longer mechanically executes tasks, but acts like an expert with both strategic (macro-level routing) and tactical (micro-level elimination) wisdom, dynamically planning the most cost-effective execution path for each query, thereby maximizing both system resource utilization and user satisfaction on a macro level.

[0098] (2) Real-time data lineage map construction and maintenance module

[0099] This module is the cornerstone of this invention, responsible for constructing and dynamically maintaining a graph structure that quantifies the health status of the data link. It aggregates various metadata and event streams from the data ecosystem in real time, constructing a directed graph—the real-time data lineage graph—with data assets as nodes, data flow relationships as edges, and edge weights quantified by real-time health indicators.

[0100] Various entities in the data ecosystem are used as nodes, including but not limited to: databases, tables, columns, ETL jobs, API endpoints, and BI dashboards.

[0101] The dependencies and flow relationships between various entities are represented as edges, for example: ETL_Job_A → Table_B (Job A writes to Table B), Table_B → API_C (API C reads from Table B);

[0102] Real-time event acquisition and calculation of health weights for each edge based on event information yield a real-time data lineage graph. Specifically, events are acquired in real-time by subscribing to message queues (such as Kafka) or polling monitoring systems. Event sources include: ETL scheduling systems (such as Airflow, DolphinScheduler): job start, success, and failure events, runtime; real-time computing engines (such as Flink, Spark Streaming): checkpoint latency, data processing throughput; database logs (such as Binlog): DDL (Data Definition Language) change events; and API gateways: API call logs, including latency and status codes.

[0103] Each edge is associated with a dynamically calculated health weight. This weight is the core of assessing the health of the data link. Its calculation formula is:

[0104]

[0105] in, This represents the health weight of an edge; a higher weight indicates a "healthier" link, making it the preferred choice for retrieval. This represents the freshness factor, used to adjust the proportion of latency and quality in the overall weight. For scenarios with high real-time requirements, It can be set to a higher value (such as 0.7); This represents the delay attenuation coefficient, used to control the nonlinear effect of delay on the weights. The larger the value, the faster the weight will drop, even with a small delay. The link quality score represents the edge, which is a comprehensive indicator that can be calculated by weighting factors such as job success rate (obtained from event source information), data quality verification pass rate, and API availability. The higher the value, the more reliable the link. This indicates the data freshness delay, measured in seconds, representing the time difference between the generation of the source data and the availability of data at the current node. The smaller the value, the fresher the data. It should be noted that: The data is obtained through regression fitting of the historical failure rate of the current node; and the system generates... Then, perform a dual-threshold linkage check: only when Seconds and Only when the condition is met will the edge be marked as a "reusable link"; otherwise, path degradation will be forcibly triggered (the system will take measures to reduce the priority or service quality of the affected path), thus forming a fundamental difference from the existing single threshold scheme. The sole source is ETL / API heartbeat events, excluding any manual backfilling or log parsing, ensuring that real-time performance cannot be replaced by static lineage.

[0106] This lineage graph is stored in a graph database (such as Neo4j, JanusGraph, NebulaGraph). Whenever a new event is received, the system updates the properties of the relevant nodes or edges in real time (e.g., ...). , ), and recalculate .

[0107] This module chooses a graph database as the storage medium, rather than a traditional relational database, because graph databases provide a natural and high-performance model for handling complex, multi-hop relationships between entities (such as data lineage). In this invention, operations such as subgraph expansion and tracing fault sources require efficient graph traversal capabilities. If a relational database were used, these operations would require complex and inefficient recursive SQL queries (recursive JOINs), which would not meet the system's real-time requirements.

[0108] (3) Intent-bloodline dynamic retrieval enhancement module

[0109] This module is used to query data source subgraphs that match user questions from real-time data lineage graphs, replacing the traditional RAG method of retrieving from static documents, and serializing the data source subgraphs into text descriptions; the specific process includes:

[0110] Intent vectorization: Using text embedding models (such as Sentence-BERT, BGE-M3) to vectorize the user's input natural language questions. Convert to a high-dimensional vector, i.e., the problem embedding vector .

[0111] Candidate node localization: In graph databases, a text embedding model is used to create vector indexes for text attributes such as table names, field names, table comments, and field comments. Perform an Approximate Nearest Neighbor (ANN) search to quickly find a batch of semantically related candidate nodes.

[0112] Subgraph Expansion and Sorting: Obtain the health weight of the upstream link (nearest neighbor node) of each candidate node, and select the candidate node corresponding to the link with the largest health weight as the final candidate node. Starting from the final candidate node, perform a weighted graph traversal on the lineage graph (such as weighted breadth-first search or a personalized PageRank algorithm) to expand a data source subgraph containing contextual relationships. During the traversal, prioritize selecting The higher edge. The final returned value. Not only is it semantically relevant, but it also represents the healthiest data access path currently available. The retrieved subgraphs... Serialized into a structured text description, containing key tables, fields, relationships between them, and comment information.

[0113] The module employs a two-stage retrieval strategy combining "vector semantic recall" and "weighted graph traversal," and its design philosophy is as follows:

[0114] 1. Addressing the efficiency issue: Graph traversal alone cannot quickly locate the initial node relevant to the user's intent from massive data assets. Therefore, we first use Approximate Nearest Neighbor (ANN) search to rapidly narrow down the range from millions of nodes to a dozen or so candidate nodes.

[0115] 2. Addressing the accuracy issue: Vector retrieval alone cannot perceive the real-time health status of the data link. Therefore, after obtaining candidate nodes, a weighted traversal is performed using the real-time lineage graph of this invention to ensure that the final selected path is the most reliable and freshest. This combination of "semantic first, lineage second" creatively balances the breadth and speed of retrieval with the quality of the final result.

[0116] (4) Traceable SQL generation and closed-loop verification module

[0117] This module is the core execution unit connecting the large model and the database, and it introduces an innovative self-healing capability. It uses the large model to generate SQL based on the text description and the user's original question, executes the SQL, and verifies the execution result. If verification fails, a closed-loop verification and repair process is initiated. The specific process is as follows:

[0118] Construct a structured Prompt, which will serialize the subgraph context from the previous step. Original user issues Several actual Few-Shot examples collected are injected into the input of large models (such as Qwen3, DeepSeek-V3, etc.), instructing the large models to output a concise, human-readable lineage TLC while generating SQL.

[0119] For example, the following TLC (Traceable Chain) clearly reveals the process by which the final metric, "Daily GMV," is generated: it originates from the most basic [order logs], undergoes real-time cleaning and processing by [Flink tasks], is then incorporated into the [order wide table] for dimensional correlation, and is finally aggregated into the [Daily GMV application table] for users to query:

[0120] TLC: ods_order_log → flink_job_dwd_order → dwd_order_wide → ads_gmv_daily

[0121] The generated SQL is executed, and the returned result set Result is quickly validated. The validation methods include: empty result validation, hash validation, or statistical validation. Empty result validation refers to checking whether the result set of the executed SQL is empty. If it is empty, the validation fails. Hash validation refers to calculating the hash value of the result of the executed SQL and checking whether it is correct. If it is incorrect, the validation fails. Statistical validation refers to checking the number of rows in the result of the executed SQL and the consistency of the content of key numerical columns. If they are inconsistent, the validation fails.

[0122] If verification fails, initiate closed-loop verification, parse the TLC attached to the SQL, and start from the final table (ads_gmv_daily) to check the health status of each node in reverse tracing the lineage (querying the real-time status of the corresponding node in the graph database). , (Attributes, etc.) are used to locate the faulty node.

[0123] Once the faulty node is located (e.g., the flink_job_dwd_order status is FAILED), the system automatically triggers a one-click rollback or node-level recalculation operation for the faulty node via API calls or execution of preset scripts, thereby completing the repair operation. Specifically:

[0124] After checking the health status of each node in reverse along the TLC, the system performs two differential operations (one of which is selected based on the actual situation):

[0125] One-click rollback: Based on the "table → job → scheduler" triplet in TLC, it automatically generates and calls the scheduling system API (such as Airflow trigger_dag), rerunning only the failed node instead of the entire link, reducing repair time by more than 60%. For example: airflow trigger_dag<dag_id> Let's rerun the failed ETL job.

[0126] Node-level recalculation: If the TLC points to a Flink SQL job in the lineage graph (the failed node is a Flink SQL node), then the stop-with-savepoint + resume-from-savepoint command is issued through the REST API to achieve minute-level incremental recalculation and avoid full-link restart.

[0127] After the repair is complete, re-execute the SQL and verify it to return the answer to the problem.

[0128] This module uses a Large Language Model (LLM) as its core for generation because of its powerful context understanding and code generation capabilities, which can handle more complex and conversational user queries than traditional template or syntax tree methods. The "Traceable Chain (TLC)" is one of the key innovations of this invention. It provides a deterministic "data receipt" for the previously "black box" SQL generation process, which can be used for automated root cause localization. This TLC acts as a bridge connecting the "SQL execution result" and the "upstream data status," and is a technical prerequisite for achieving subsequent closed-loop verification and self-healing functions.

[0129] (5) Zero-sample memory reuse module

[0130] This module constructs memory units from queries generated through traceable SQL generation and verified by a closed-loop validation module, and dynamically evaluates these memory units. Based on the evaluation results, it performs semantic reuse and intelligent eviction. Specifically, when encountering a semantically similar new problem, the module uses a decision-making process integrated with a multi-armed machine (MAB) to intelligently determine whether to directly reuse historical results or execute the entire query chain, and dynamically evicts inefficient cached queries based on the results. It provides the system with "memory" capabilities, reducing costs and increasing efficiency for high-frequency, repetitive queries. Through a complete lifecycle management mechanism including creation, semantic reuse, dynamic evaluation, and intelligent eviction, it ensures that the query memory is not only used efficiently but also continuously self-optimizes.

[0131] Constructing memory units:

[0132]

[0133] in, These are question vectors, which are mathematical and vectorized representations of the user's original natural language question (such as "Check yesterday's GMV") after processing by a text embedding model. Their function is as a "semantic index" for the memory database. They allow the system to match queries with similar meanings but different wording by calculating the cosine similarity between vectors, achieving cache hits at the "fuzzy" or "semantic" level.

[0134] This is the SQL statement text, which serves as an "executable entity" for caching. Once the cache is hit, the system can directly reuse this SQL code, thereby bypassing the expensive steps of calling large models for analysis and generation, and achieving millisecond-level query response.

[0135] This is a traceable chain of text, a concise text describing the core data chain upon which the SQL query depends (e.g., ods_log → dwd_order → ads_gmv). Its purpose is to provide a "lineage snapshot" and "audit credentials" for the query. Even when rapidly reusing the cache, the system can know the data source of the result, greatly improving system observability and troubleshooting efficiency.

[0136] It is the result hash value, which is... The result set returned after execution is hashed (e.g., using SHA256) to obtain a unique "digital fingerprint." Its purpose is as a low-cost, high-efficiency data consistency verification mechanism. It ensures that even if the query logic is the same, if the underlying data changes, the system can detect the change by comparing the different hash values, thus avoiding the return of outdated cached data.

[0137] This is the timestamp of when the memory unit was created or last successfully verified. Its purpose is to manage the time-to-live (TTL) validity of the cache. The system can use this to determine whether a cache has "expired" due to excessive time, which is especially important for time-sensitive queries (such as "today" or "this week").

[0138] This is the upper confidence bound score of the memory cell. This is the core of the MAB algorithm, representing the "estimated potential" of the cache. The higher the score, the more worthwhile it is to be "utilized".

[0139] This is the number of times the memory cell was accessed (or evaluated as a candidate arm). It is used to calculate the UCB value and is another key input to the MAB algorithm.

[0140] Whenever a query passes the "closed-loop verification module" and successfully returns a result, its corresponding core data quintuple is constructed into a memory unit and initialized. and Then it is written to the module's memory database (such as Milvus, Faiss, Qdrant).

[0141] The design concept of this memory unit aims to overcome the two major shortcomings of traditional caches and build a truly intelligent and reliable "memory" system:

[0142] 1. The leap from "exact matching" to "semantic matching":

[0143] Traditional caches (such as Redis KV cache) can only be hit when the user inputs the exact same string. This is almost ineffective in varied natural language scenarios. This design introduces... (Question vectors) elevate the matching benchmark from "text" to "semantics". The system no longer cares about "how the user asks" but rather "what the user asks", which results in an order-of-magnitude improvement in cache hit rate and usability.

[0144] 2. The evolution from "blind trust" to "proactive verification":

[0145] Traditional caches, once stored, unconditionally trust their content, which can easily lead to returning "dirty data" after the source data has been updated. This invention introduces... The hash value of the result serves as a "data seal." Each time a cache hit occurs, the system re-executes the SQL and verifies the hash value at extremely low cost. This simple action enables real-time, low-cost verification of data freshness, ensuring the reliability of cached results from a fundamental mechanism and solving the consistency problem between the cache and the data source.

[0146] This invention abandons the simple, static "hit-and-play" rule of traditional caching, and integrates reuse decisions, performance evaluation, and cache eviction into a dynamic learning closed loop based on MAB; such as Figure 3 As shown, the process of dynamically evaluating memory units and performing semantic reuse and intelligent elimination based on the evaluation results includes:

[0147] The user's question is embedded to obtain a question embedding vector. ;

[0148] Search the memory database of the zero-sample memory reuse module to find the k non-expired memory units that are most similar to the problem vector (cosine similarity can be used) and use them as k cache arms;

[0149] The system will this A "cache arm" and a special "baseline arm" (i.e., bypassing the cache to execute the complete NL2SQL chain) together constitute a Arm slot machine; using the first-level MAB decision to select the UCB value from k+1 arms ( The highest-performing arm is chosen as the execution result. This decision-making process elegantly balances "utilization" (selecting the historically best-performing arm) and "exploration" (giving opportunities to underperforming or new arms); specifically: if a "cached arm" is selected, then its execution is performed. And verify If the "baseline arm" is selected, the complete NL2SQL chain is executed; regardless of which arm is selected, the reward for that arm is updated after execution based on the actual cost and latency (the calculation formula is the same as the reward formula for the first-level MAB decision; in this module, ...). and It measures the micro-performance overhead of executing a single cache item or a single complete chain execution, with the aim of weeding out inferior assets within the cache pool. The UCB value of this arm is updated based on the reward.

[0150] The UCB algorithm is a method for solving the MAB problem. It determines which arm to choose next by maintaining a confidence upper bound for each arm. This confidence upper bound is calculated based on the arm's historical rewards and the number of times it has been selected. Specifically, for each arm i, its UCB value can be calculated using the following formula:

[0151]

[0152] in, It is the average reward estimate for the i-th arm. This indicates the number of times the i-th arm was selected. This represents the total number of attempts. This represents a parameter that controls the degree of exploration.

[0153] If the UCB value of a cache arm is less than the preset threshold N times consecutively (which proves that the cache has become "rigid" and its benefits are not even as good as regeneration), the system will trigger an asynchronous eviction mechanism to completely remove the cache arm from the memory database; otherwise, it will determine whether the hash value of the cache arm has changed after this execution. If there is no change, the memory unit will be retained; if there is a change, the hash value and timestamp of the memory unit will be updated.

[0154] This module uses a vector database instead of a traditional key-value cache (such as Redis). Its core advantage lies in its ability to achieve "semantic caching" rather than "exact text matching caching." Traditional caches only hit when users input identical questions, while this invention, through vector similarity search, can identify queries that are "semantically equivalent but different in expression" (e.g., "Check yesterday's GMV" and "What was yesterday's total sales?"), greatly improving the cache hit rate. Simultaneously, the introduction of result hash verification is a low-cost and efficient data consistency guarantee mechanism, ensuring that even if the query logic is the same, the system can detect updates to the underlying data, avoiding the return of outdated results.

[0155] In some preferred embodiments of the present invention, the system further includes an exception triggering function, the implementation steps of which include:

[0156] The real-time data lineage graph construction and maintenance module monitors each data link in the real-time data lineage graph. When an anomaly is detected in the quality attributes of a data link, an analysis intent is generated.

[0157] The intent-based dynamic retrieval enhancement module retrieves subgraphs related to abnormal links from the real-time data lineage graph based on the analysis intent.

[0158] The traceable SQL generation and closed-loop verification module generates comparative analysis SQL based on the subgraph and analysis intent; the SQL is executed to obtain the analysis results.

[0159] The system integrates all information into structured alarm messages and sends them to the user.

[0160] The following are examples of implementing the present invention:

[0161] Example 1:

[0162] Real-time financial fraud monitoring scenarios – from query failures to automatic recovery

[0163] Scenario Description: In the risk control center of a large bank, a risk analyst urgently needs to respond to a new wave of fraud attacks. He sends a natural language query to the data analytics platform: "Give me the total amount and number of credit card fraud orders involving virtual goods transactions that occurred in the xx region in the past 30 minutes." This query requires correlating transaction logs, user profile tables, and risk control rule result tables, which were generated by different real-time and near-real-time ETL jobs.

[0164] Implementation process:

[0165] Step 1: Cost-Latency Aware Smart Router (Module 1):

[0166] The system has received the question. Since this is the first query for a completely new question, there is no semantically matching cache in the zero-shot memory (module 4);

[0167] The second layer of the routing module, MAB, needs to choose between ROUTE_DB (generating answers through database queries) and ROUTE_API (directly reasoning for answers through LLM). Based on historical models, for analytical problems requiring precise aggregation calculations, the average reward (lower cost, higher information certainty) of the ROUTE_DB path is significantly higher than that of the ROUTE_API path.

[0168] Decision: MAB executes the "Exploitation" strategy, determines that ROUTE_DB is the current optimal macro path, and forwards the request to subsequent modules for processing.

[0169] Step 2: Enhanced Intent-Kinship Dynamic Retrieval (Module 2):

[0170] Intent understanding and semantic recall: The system receives a question and converts it into a vector using a text embedding model. .pass An ANN search was performed in the graph database, which initially retrieved two potentially relevant core indicator tables: ads_fraud_monitor_30min (a wide table for 30-minute fraud monitoring) and ads_fraud_summary_daily (a daily fraud overview table).

[0171] Real-time lineage assessment: The system immediately queries the real-time data lineage graph (module 1) to obtain the real-time health of the upstream data links in these two tables: for ads_fraud_monitor_30min, the edge weights of its upstream critical job flink_job_dwd_fraud_rt. The link freshness score is 0.95 (data freshness Δt = 25 seconds, link quality Q = 0.98), indicating a healthy link status. However, for ads_fraud_summary_daily, the edge weight of its upstream ETL job is only 0.21 (data freshness Δt = 2 hours, link quality Q = 0.99), indicating that the data is severely outdated.

[0172] Decision: Based on the dynamic weights of the lineage graph, the system determines that ads_fraud_monitor_30min is the healthiest and most timely choice under the current intent, and expands it to a health data source subgraph containing nodes such as ods_card_txn_rt and dwd_fraud_rt.

[0173] Step 3: Traceable SQL Generation (Module 3)

[0174] Prompt Construction and Generation: The system will serialize the G... k (Including table structure, field comments, and relationships) is injected into the Llama3 model's Prompt along with the user's question, with an explicit instruction: "Please generate SQL and output its core data lineage (TLC)"; then the LLM outputs the SQL.

[0175] Step 4: Closed-loop verification and self-healing (Module 3)

[0176] Initial Execution and Anomaly Detection: The system executes the above SQL, but the database returns {sum: null, count: 0} after 3 seconds. This is immediately identified as an anomaly (because according to historical statistical models, the probability of zero fraudulent transactions within 30 minutes in a certain financial region is less than 0.01%, which is highly inconsistent with common business sense).

[0177] Fault location: The closed-loop verification process was triggered. The system analyzes the TLC and begins reverse tracing:

[0178] Checking the ads_fraud_monitor_30min node: A query in the graph database shows that its Δt = 25 seconds, indicating a normal status.

[0179] Inspecting the previous hop, the `flink_job_dwd_fraud_rt` node, revealed its real-time attribute as `{status: FAILED, failure_time: '2025-08-06 10:32:15', reason: 'Upstream Kafkatopic partition 3 consumption lag exceeded threshold'}`. The source of the fault was precisely located.

[0180] Automatic Repair: Based on preset repair rules, the system matches the repair action for "Kafka consumption latency". It sends a command to the Airflow system via API call: `airflow trigger_dag dwd_fraud_rt_recovery_dag --conf '{"kafka_partition_to_reset": 3}'`. Simultaneously, it sends a notification to the risk control and operations team: "Job dwd_fraud_rt has failed due to upstream data backlog, and the automatic recovery process has been triggered."

[0181] Successful response: Approximately 2 minutes later, the recovery job executed successfully, and the data link was restored. The system automatically re-executed the previous SQL, this time successfully returning the result: {sum: 127540.00, count: 18}. Finally, this correct result was presented to the analyst.

[0182] In this example, using the traditional method, the risk analyst would see an empty result and need to contact the IT department. IT personnel might spend 1-2 hours checking logs, inspecting ETL, and contacting data engineers to locate and fix the problem. This entire process is time-consuming and severely impacts the timeliness of risk control decisions. In contrast, this invention automates the entire process—from problem discovery to automatic location, triggering repair, re-querying, and returning the correct result—in approximately 3 minutes, achieving full automation. This reduces risk control response time by over 95% and avoids business misjudgments caused by data errors.

[0183] Example 2:

[0184] E-commerce promotion real-time command center scenario – MAB-driven extreme optimization of cost and efficiency

[0185] Scenario Description: During a major sales promotion, the operations director, in front of the control panel, frequently queries the same core question via voice assistant: "What is the current real-time total GMV (Gross Merchandise Volume)?" This data can be obtained in three ways: 1) Directly querying the core transaction database (high load, impacting online transactions); 2) Calling a per-transaction ($0.02 / transaction) aggregated data API; 3) Reading a near real-time materialized view within the system (refreshed every minute).

[0186] Implementation process:

[0187] 1. First query - 10:00:00 AM

[0188] Scenario: The operations director asks for the first time: "What is the current real-time total GMV?"

[0189] Step 1: Macro-routing decision (Module 2 - Layer 2 MAB)

[0190] Arm definition: At this time, MAB has three "macro arms" to choose from: ROUTE_CACHE (caching arm, but currently empty), ROUTE_DB (database arm), and ROUTE_API (API arm).

[0191] Status assessment: Module intelligence gathering: L_db=92% (high load), Cost_API=$0.02.

[0192] MAB Decision: Despite high DB load, historical data shows that ROUTE_DB generally offers better overall performance (cost-effectiveness) than the more expensive ROUTE_API. MAB employs a "utilization" strategy and selects ROUTE_DB.

[0193] Step 2: Execution and Feedback

[0194] The system generated and executed SQL queries targeting the core transaction database using modules 3 and 4. Due to high database load, the query took 2.5 seconds.

[0195] Reward Calculation: The Actual Latency of this action is very high, resulting in a low calculated "Reward" value.

[0196] MAB Status Update: The UCB (Confidence Upper Bound) value of the ROUTE_DB arm has been lowered due to this low reward.

[0197] Step 3: Write to the memory bank (Module 5)

[0198] Once the query is successful, the system stores this session (question vector, SQL, TLC, result hash) as a new "memory unit" in the vector library. This memory unit becomes a new "micro-arm" of the first-level MAB.

[0199] 2. Second query - 10:01:00 AM

[0200] Scenario: The operations director changed the question: "What is the total sales figure now?"

[0201] Step 1: Macro-routing decision (Module 1 - Layer 2 MAB)

[0202] MAB Decision: At this point, the expected reward for the ROUTE_CACHE arm (because content has just been written) becomes attractive. The second-level MAB chooses the "Exploration" ROUTE_CACHE path.

[0203] Step 2: Micro-caching decision (Module 5 - Level 1 MAB)

[0204] Arm definition: When a request enters module 4, its MAB faces two "micro arms": the "GMV cache arm" that was just created and a "baseline arm" (representing retracing the complete DB query chain).

[0205] MAB Decision: The "GMV Cache Arm" is a new arm with a high UCB value (high exploration value), so MAB chooses to "utilize" this cache arm.

[0206] Execution and Verification: The system hit the cache arm through semantic search. It executes its SQL (SELECT gmv FROM mv_gmv_1min;) and performs a result hash verification. At this point, the materialized view has been updated, returning the new value "1,582,112,345.00 yuan". The system calculates the hash of the new result and finds it inconsistent with the result_hash stored in the memory unit.

[0207] Decision and Return: The system determines that the data has been updated and directly returns the new, correct result to the user, taking 60 milliseconds. Simultaneously, the result_hash of that unit in the memory is updated asynchronously.

[0208] Rewards and Updates: This action resulted in extremely low Actual_Latency and Actual_Cost, earning a very high "reward". This high reward also updated the UCB values ​​of the first-level MAB's "GMV cache arm" and the second-level MAB's ROUTE_CACHE arm, significantly improving their "reputation".

[0209] 3. Third query - 10:02:30 AM, an unexpected situation occurred.

[0210] Scenario: The operations director queries again. However, at this time, the ETL job used to generate the GMV materialized view unexpectedly fails, and the health of the corresponding link in the lineage graph (module 2) is affected. A sudden drop.

[0211] Step 1: Macro and Micro MAB Decision Making

[0212] Based on the high rewards from the previous event, both the second-level MAB and the first-level MAB will still prioritize the ROUTE_CACHE path and the "GMV cache arm".

[0213] Step 2: Perform dynamic adjustments with MAB

[0214] Execution and Verification: The system executes the cached SQL, but because the upstream ETL has failed, the query returns outdated data. As a result, hash verification fails multiple times (or, better yet, TLC verification finds that the upstream link is unhealthy).

[0215] Reward Calculation: Each failed verification is considered a failed execution, and the system calculates a very low "reward" for the "GMV Cache Arm".

[0216] MAB Status Update: After several failures, the UCB value of the "GMV Cache Arm" in the first-level MAB rapidly decayed and soon fell below the UCB value of the "Base Arm".

[0217] Strategy Switching: In the next decision-making phase, the first-layer MAB will automatically abandon the "GMV cache arm" and instead select the "baseline arm," meaning it will re-execute the complete DB query chain, thus ensuring the accuracy of the results. Simultaneously, the second-layer MAB, due to the continuous low rewards generated by the ROUTE_CACHE path, will begin to reduce its UCB value and consider re-exploring other paths such as ROUTE_API.

[0218] Comparison of this invention with traditional methods:

[0219] Traditional methods rely on static rules. This either constantly overloads the database, uses expensive APIs, or unknowingly returns erroneous cached data after ETL failures.

[0220] Querying via cache arms not only helps verify the effectiveness of caching strategies but also provides a faster and more efficient data access path for the future. Furthermore, this approach helps the system better handle high-concurrency requests while reducing pressure on the database. This invention, through a two-layer MAB dynamic learning mechanism, enables the system not only to find the initial optimal path but, more importantly, to automatically and quickly adapt when system states (DB load, ETL health) change. It discovers and utilizes efficient paths through low-cost "exploration," and automatically switches to suboptimal solutions through continuous feedback learning after a path fails, demonstrating extremely high adaptability, robustness, and cost efficiency.

[0221] Example 3:

[0222] Scenario Description: In a large automotive parts manufacturing plant, a critical automated production line (Line 3) has hundreds of sensors that transmit data such as temperature, pressure, and vibration frequency in real time. This data is fed into a data platform to monitor the health of the production line and the product yield rate.

[0223] Implementation process:

[0224] Step 1: Active monitoring of real-time lineage chart (Module 2)

[0225] Normal state: Module 1 of this invention continuously monitors thousands of data links from various sensors to downstream aggregation tables (such as dws_production_line_metrics_1min). Under normal circumstances, the quality Q value of all links is above 0.99.

[0226] Proactive anomaly detection: At 14:30:00 on August 6, 2025, Module 1 detected 15 temperature sensor data streams from "Production Line 3 - Heat Treatment Unit". The quality attribute Q of the corresponding edge dropped sharply from 0.99 to 0.65 within 5 minutes. The reason was that the gateway device upstream of the sensor experienced intermittent packet loss, resulting in discontinuous data points.

[0227] Triggering linkage: This "regional collapse" in the lineage diagram is identified by the system as a high-priority abnormal event and automatically triggers the next step.

[0228] Step 2: Proactive Attribution Analysis Based on Anomalies (Modules 3+4)

[0229] Generate analysis intent: Based on the context of the anomaly (node: production line 3 - heat treatment unit, attribute: data quality deterioration), the system automatically generates an analysis intent: "Investigate whether the product yield of 'production line 3' is affected by the data anomaly of the 'heat treatment unit' and compare it with the healthy 'production line 2'."

[0230] Dynamic retrieval and SQL generation:

[0231] Based on this intent, Module 2 retrieves the core indicator table most relevant to "product yield" and "production line 3 / 2" from the lineage graph, such as ads_product_yield_hourly.

[0232] Module 3 receives this subgraph and analysis intent, and proactively generates an SQL statement for comparative analysis.

[0233] Step 3: Generate actionable insights and alerts

[0234] Execution and Verification: The system executed the SQL and obtained the result: {Line-02: 0.985, Line-03: 0.821}. The result clearly shows that the yield rate of production line 3 is significantly lower than that of the normal production line 2.

[0235] Generate human-readable insights: The system integrates all information into a structured alert and insight, which is then pushed to the user interface.

[0236] [Level 1 Alert: Abnormal yield rate for production line 3]

[0237] Phenomenon: The average yield of production line 3 dropped to 82.1% in the past hour (compared to 98.5% for production line 2 in the control group).

[0238] Root cause (automatic attribution): The system traced the temperature sensor data quality upstream to the "heat treatment unit," where a significant drop occurred at 14:30.

[0239] Recommended action: Please immediately check the sensors and network transmission equipment of heat treatment unit No. 3 on production line.

[0240] Comparison of this invention with traditional methods:

[0241] In the traditional way, workshop supervisors might not discover the decline in yield rate until they see the reports several hours later, or even the next day. By then, a large number of defective products have already been produced, and a lot of manual time is required to check for possible mechanical, electrical, and network problems one by one, making it difficult to attribute the cause.

[0242] Within minutes of a problem occurring, this invention not only proactively identifies the root cause at the data level but also automatically performs correlation analysis of the business impact (decreased yield rate), providing clear attribution conclusions and action recommendations. This reduces the problem discovery and repair cycle from "hours / days" to "minutes," significantly minimizing production losses and demonstrating a leap from a "data platform" to an "intelligent decision-making hub."

[0243] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic RAG-NL2SQL system based on real-time data lineage graphs, characterized in that, include: The module includes a real-time data lineage graph construction and maintenance module, an intent-lineage dynamic retrieval enhancement module, a traceable SQL generation and closed-loop verification module, a zero-sample memory reuse module, and a cost-delay-aware intelligent routing module. The cost-latency-aware intelligent routing module receives user questions and performs macro-level meta-decision to obtain the optimal path to answer user questions and calls relevant modules to execute them. The real-time data lineage graph construction and maintenance module is used to construct and dynamically maintain the real-time data lineage graph; The intent-line dynamic retrieval enhancement module is used to query the data source subgraph that matches the user's question from the real-time data lineage graph and serialize the data source subgraph into a text description; The traceable SQL generation and closed-loop verification module uses a large model to generate SQL based on text description and user's original question, executes the SQL and verifies the execution result. If the verification fails, the closed-loop verification and repair process is initiated. The zero-sample memory reuse module is used to construct memory units from queries that have been verified by the traceable SQL generation and closed-loop verification module, and to dynamically evaluate the memory units; based on the evaluation results, semantic reuse and intelligent elimination are performed.

2. The dynamic RAG-NL2SQL system based on real-time data lineage graphs according to claim 1, characterized in that, The process by which the cost-latency-aware intelligent routing module receives user questions and makes macro-level meta-decision includes: The cost-latency-aware intelligent routing module calculates the cost of three paths in real time, respectively. Database execution cost of the path API call cost of the path and The cost of cached lookup for a path; The path is to retrieve the answer from the zero-sample memory; The path involves generating SQL via LLM and executing it in an internal database to retrieve the answer. The approach involves obtaining the answer through direct reasoning using LLM; The second-level MAB decision method calculates the reward scores for the three paths based on their costs, and selects the path with the highest reward score as the best path to answer the user's question.

3. The dynamic RAG-NL2SQL system based on real-time data lineage graphs according to claim 2, characterized in that, The formula for calculating the bonus score is: ； in, Indicates the reward score. Indicates cost weight. Indicates the delay weight. Represents the cost of the path. Indicates the path delay.

4. The dynamic RAG-NL2SQL system based on real-time data lineage graphs according to claim 1, characterized in that, The process of constructing and maintaining the real-time data lineage graph by the module includes: taking various entities in the data ecosystem as nodes and the dependencies and flow relationships between various entities as edges; acquiring events in real time and calculating the health weight of each edge based on the event information to obtain the real-time data lineage graph.

5. The dynamic RAG-NL2SQL system based on real-time data lineage graphs according to claim 4, characterized in that, The formula for calculating the health weight of an edge is: ； in, Represents the health weight of the edge. Represents the freshness factor. This represents the delay attenuation coefficient. The link quality score represents the edge. This indicates a delay in data freshness.

6. The dynamic RAG-NL2SQL system based on real-time data lineage graphs according to claim 1, characterized in that, The process by which the intent-line dynamic retrieval enhancement module queries the data source subgraph that matches the user's question from the real-time data lineage graph includes: The user's question is embedded to obtain a question embedding vector; a vector index is built based on the real-time data lineage graph. Based on the question embedding vector, an approximate nearest neighbor search is performed in the vector index of the real-time data lineage graph to obtain multiple candidate nodes; Obtain the health weight of the upstream link of each candidate node, and select the candidate node corresponding to the link with the largest health weight as the final candidate node; Starting from the final candidate node, perform a weighted graph traversal on the lineage graph to obtain the data source subgraph.

7. The dynamic RAG-NL2SQL system based on real-time data lineage graphs according to claim 1, characterized in that, The traceable SQL generation and closed-loop verification module verifies the results of SQL execution in the following ways: empty result verification, hash verification, or statistical verification. Empty result verification refers to determining whether the result set of the executed SQL is empty; if it is empty, the verification fails. Hash verification refers to calculating the hash value of the executed SQL result and determining whether it is correct; if it is incorrect, the verification fails. Statistical verification refers to checking the consistency of the number of rows and the content of key numerical columns in the executed SQL result; if they are inconsistent, the verification fails.

8. The dynamic RAG-NL2SQL system based on real-time data lineage graphs according to claim 1, characterized in that, The process of initiating the closed-loop verification and repair process by the traceable SQL generation and closed-loop verification module includes: When verification fails, initiate closed-loop verification, parse the TLC attached to the SQL, and check the health status of each node in reverse order along the lineage starting from the final table to locate the faulty node. Automatically trigger a one-click rollback or node-level recalculation operation for the faulty node to complete the repair; Re-execute the SQL and verify it to return the answer to the question.

9. A dynamic RAG-NL2SQL system based on real-time data lineage graphs according to claim 1, characterized in that, The zero-sample memory reuse module dynamically evaluates memory units, and the process of semantic reuse and intelligent elimination based on the evaluation results includes: The user's question is embedded to obtain a question embedding vector; Search the memory database of the zero-sample memory reuse module to find the k non-expired memory units that are most similar to the problem vector and use them as k cache arms; The complete NL2SQL link is used as the baseline arm. The first-level MAB decision selects the arm with the highest UCB value among k+1 arms as the execution result. After execution, the reward of the arm is updated according to the actual cost and delay; the UCB value of the arm is updated according to the reward. If the UCB value of a cache arm is less than the preset threshold N times consecutively, the cache arm is removed from the memory database; otherwise, it is determined whether the hash value of the cache arm has changed after this execution. If it has not changed, the memory unit is retained; if it has changed, the hash value and timestamp of the memory unit are updated.

10. A dynamic RAG-NL2SQL system based on real-time data lineage graphs according to claim 1, characterized in that, The system also includes an exception triggering function, which is implemented through the following steps: The real-time data lineage graph construction and maintenance module monitors each data link in the real-time data lineage graph. When an anomaly is detected in the quality attributes of a data link, an analysis intent is generated. The intent-based dynamic retrieval enhancement module retrieves subgraphs related to abnormal links from the real-time data lineage graph based on the analysis intent. The traceable SQL generation and closed-loop verification module generates comparative analysis SQL based on the subgraph and analysis intent; the SQL is executed to obtain the analysis results. The system integrates all information into structured alarm messages and sends them to the user.

Images