A large model-based data bloodline analysis method, system, device and storage medium

Abstract

The application provides a data blood relationship analysis method, system and device based on a large model and a storage medium, and belongs to the technical field of big data analysis applications. The method comprises collecting original SQL data, processing the collected original SQL data to obtain standard SQL data, vectorizing the standard SQL data to form a vector knowledge base, constructing an interactive model Agent based on a large model according to the vector knowledge base, performing data blood relationship analysis on the vector knowledge base by using the interactive model Agent based on a large model, and generating a data problem diagnosis result. The data blood relationship analysis method can not only help data warehouse personnel understand the blood relationship of data, but also actively locate and analyze data problems and give specific solutions, thereby improving the work efficiency of data application development.

Description

Technical Field

[0001] This invention relates to the field of big data analytics application technology, and in particular to a data lineage analysis method, system, device, and storage medium based on a large model. Background Technology

[0002] Data lineage refers to the natural relationships, much like human blood ties, that form between data throughout their entire lifecycle—from generation, processing, manipulation, fusion, and flow to their eventual destruction. It involves not only the physical flow of data but also their logical relationships and transformation processes. Data lineage analysis is a core function of metadata management and data governance tools. By establishing lineage relationships between data, it's possible to analyze whether changes in upstream data affect related downstream data. Furthermore, by establishing relationships between technical and business metadata, lineage analysis can reveal data flow between different business products and their business relationships. Analyzing data lineage leads to a better understanding and use of data. For example, CN118467175A discloses a data lineage analysis method.

[0003] However, current data lineage analysis can only list the upstream and downstream relationships between data, that is, the relationships between tables, and cannot answer questions related to lineage or provide solutions. Summary of the Invention

[0004] To address the above issues, this invention provides a data lineage analysis method, system, device, and storage medium based on a large model, which helps data warehouse developers quickly locate data problems through lineage analysis Q&A and improves the efficiency of data application development.

[0005] This invention provides a data lineage analysis method based on a large model, comprising: Collect raw SQL data and process the collected raw SQL data to obtain standard SQL data; The standard SQL data obtained after vectorization processing forms a vector knowledge base; Based on the vector knowledge base, construct an interaction model Agent based on a large model; The large-model-based interaction model Agent is used to perform data lineage analysis on the vector knowledge base and generate data problem diagnosis results.

[0006] As a further improvement of the present invention, the step of collecting raw SQL data and processing the collected raw SQL data to obtain standard SQL data includes: Collect SQL data to obtain raw SQL data; The original SQL data is disassembled and extracted; Preprocess the decomposed and extracted SQL data; Extract information from the preprocessed SQL data; Save the extracted SQL data into a specific structure.

[0007] As a further improvement of the present invention, the process of disassembling and extracting the original SQL data includes disassembling multiple SQL segments into a single SQL statement and extracting only the insert statement SQL.

[0008] As a further improvement of the present invention, the preprocessing of the disassembled and extracted SQL data includes: deleting remarks and replacing line breaks. Replace tab characters "\t" and remove extra spaces.

[0009] As a further improvement of the present invention, the information extraction of the preprocessed SQL data includes: extracting all table names, deleting duplicate table names, and removing the first table name after the insert_table statement and saving it separately as the target table, while saving the others as sub-tables.

[0010] As a further improvement of the present invention, the step of saving the extracted SQL data into a specific structure includes saving the extracted SQL data into {"lineage": target table, "sub-table structure": sub-table}.

[0011] As a further improvement of the present invention, the step of constructing an interaction model agent based on a large model according to the vector knowledge base includes using langchain to embed the vector knowledge base and constructing an interaction model agent based on a large model.

[0012] This invention provides a data lineage analysis system based on a large model, comprising an SQL data collection and processing module, a vector knowledge base formation module, an interaction model agent construction module, and a lineage analysis and problem diagnosis module, wherein: The SQL data collection and processing module is used to collect raw SQL data and process the collected raw SQL data to obtain standard SQL data. The vector knowledge base forming module is used to form a vector knowledge base from the standard SQL data obtained after vectorization processing. The interaction model Agent construction module is used to construct an interaction model Agent based on a large model according to the vector knowledge base; The lineage analysis and problem diagnosis module is used to perform data lineage analysis on the vector knowledge base using an interaction model Agent based on a large model, and generate data problem diagnosis results.

[0013] The present invention provides a computer electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described data lineage analysis method based on a large model.

[0014] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described data lineage analysis method based on a large model.

[0015] This invention provides a data lineage analysis method, system, device, and storage medium based on a large model. It collects raw SQL data and processes it to obtain standard SQL data. The standard SQL data, after vectorization, forms a vector knowledge base. Based on the vector knowledge base, an interaction model agent based on the large model is constructed. The interaction model agent performs data lineage analysis on the vector knowledge base and generates data problem diagnostic results. Based on large language model content generation technology, this invention helps data warehouse developers quickly locate data problems through lineage analysis question answering, improving the efficiency of data application development. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the data lineage analysis method based on a large model according to an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of a specific analysis process for a data lineage analysis method based on a large model according to an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the structure of data lineage analysis based on a large model according to an embodiment of the present invention.

[0019] Figure labeling: 10, SQL data collection and processing module; 20, vector knowledge base formation module; 30, interaction model agent construction module; 40, lineage analysis and problem diagnosis module. Detailed Implementation

[0020] The following describes specific embodiments and appendices. Figure 1-3 The invention is described in detail so that those skilled in the art can more fully understand its purpose, features and effects.

[0021] Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. In the event of any discrepancy between the definitions of terms in this invention and their commonly understood meaning by one of ordinary skill in the art to which this invention pertains, the definitions provided herein shall prevail.

[0022] In today's big data environment, data is experiencing explosive growth. To better govern big data, extract value from it, ensure data quality, and improve data credibility, it is crucial to do so. Data lineage analysis can ensure the smooth completion of these processes. By tracing the relationships between data points, data lineage analysis clearly demonstrates the entire transformation process of data from its source to its current form, making the data traceable and verifiable. This significantly improves data quality and addresses issues such as uncertainty and unreliability. Data lineage analysis allows users to clearly see the processing of data entities, facilitating data verification and visually demonstrating the data's lineage relationships through the data lineage chain.

[0023] This invention improves existing data lineage analysis by providing a data lineage analysis method, system, device, and storage medium based on a large model, enabling data warehouse developers to quickly locate data problems.

[0024] Example 1 As a specific embodiment of the present invention, this embodiment provides a data lineage analysis method based on a large model, referring to... Figure 1 , Figure 2 The specific steps are as follows: S100. Collect raw SQL data and process the collected raw SQL data to obtain standard SQL data; S200. The standard SQL data obtained after vectorization processing forms a vector knowledge base; S300. Based on the vector knowledge base, construct an interaction model Agent based on a large model; S400. The Agent based on the large model is used to perform data lineage analysis on the vector knowledge base and generate data problem diagnosis results.

[0025] The data lineage analysis method based on large models provided by this invention is based on large language model technology and can help data warehouse engineers quickly analyze data lineage issues.

[0026] Specifically, S100 collects raw SQL data and processes the collected raw SQL data to obtain standard SQL data, including: S110. Collect SQL data and obtain raw SQL data. The SQL data includes table and field names, as well as the relationships between tables, which is essential information for lineage analysis. The raw SQL data is obtained by collecting table creation SQL data from the codebase.

[0027] S120. Decompose and extract the original SQL data. When breaking down the original SQL data, multiple SQL statements are separated into single SQL statements. A single SQL statement is easier to locate the problem compared to multiple SQL statements. The single SQL statement is used for subsequent content extraction. When multiple SQL statements are mixed together, the program cannot recognize the information in the tables and fields for regular expression extraction; therefore, it is necessary to break down the multiple SQL statements into single SQL statements.

[0028] When extracting SQL data, only INSERT statements are extracted. Ideally, the SQL statement should be extracted based on whether it begins with an INSERT statement.

[0029] The above processing can make the obtained SQL more accurate, reduce data volume, and reduce unnecessary data.

[0030] S130. Preprocess the disassembled and extracted SQL data, including: Delete the remarks; specifically, remove all characters after the "--" symbol. Replace newline character " Specifically, all newline characters " Replace " with a space; Replace the tab character "\t" with a space; Remove extra spaces; specifically, replace consecutive spaces with a single space.

[0031] The above processing eliminates useless data carried in the original SQL data, reduces data redundancy, and improves the quality of SQL data.

[0032] S140. Extract information from the preprocessed SQL data, including: Extract all table names; Delete duplicate table names; Remove the first table name after the insert_table statement and save it separately as "target table", and the others as sub-tables.

[0033] By deduplicating data, we can maintain the uniqueness of table names and avoid errors in later data processing.

[0034] S150. Organize the target table and sub-tables saved in S140 into a specific structure. Specifically, the specific structure is: {"lineage": target table, "sub-table structure": sub-table}.

[0035] By saving the target tables and sub-tables in the processed SQL data as a specific structure, making them more compliant with post-processing standards, we can improve the efficiency of subsequent data use and facilitate the analysis of data lineage issues.

[0036] By processing the collected raw SQL data, removing junk data, extracting key information, and forming a usable data structure, standard SQL data is obtained.

[0037] The data collection and processing steps are used to collect SQL data, process it, output table field relationships, and form the standard SQL data required for building a vector database (vector knowledge base).

[0038] The standard SQL data obtained after S200 vectorization processing forms a vector knowledge base by embedding the processed standard SQL data into the Data Connection component of langchain. When a question is subsequently posed, the corresponding knowledge from the vector knowledge base is retrieved, fed into the large model, processed, and interpreted to form the answer.

[0039] LangChain technology provides a powerful set of tools and interfaces that simplify the entire process from raw data collection to the formation of a vector knowledge base. Its Data Connection component facilitates easier connection to different data sources, automating data cleaning and transformation steps and accelerating data processing. By transforming SQL data into vector form, LangChain enables better understanding and manipulation of this data in a high-dimensional space. This vectorized representation not only supports more precise data querying and retrieval but also allows for the application of sophisticated machine learning models for data analysis.

[0040] S300 constructs an interaction model agent based on a large model according to the vector knowledge base, including: using langchain to embed the vector knowledge base and constructing an interaction model agent based on a large model.

[0041] By using langchain technology and embedding the vector knowledge base formed in S200, a lineage analysis big data model agent is built that can interact with data warehouse personnel in natural language, providing users with a more natural and efficient way to interact with data.

[0042] S400 utilizes the large-model-based interaction model Agent to perform data lineage analysis on the vector knowledge base and generates data problem diagnosis results, including constructing a CoT (Coordinated Inquiry) system based on the large model, configuring an intent recognition large model, configuring an entity recognition large model, and configuring a target setting large model. By constructing the CoT system based on the large model, analytical capabilities and accuracy can be enhanced.

[0043] The large intent recognition model is used to understand the real purpose behind a user's query or command; the large entity recognition model is used to identify data-related entities, such as table names; the large goal-setting model is used to clarify the specific objective to be achieved or the problem to be solved through data lineage analysis.

[0044] Chain of Thought (CoT) is a problem-solving method through step-by-step reasoning. By constructing a CoT system, the interactive model Agent can more effectively understand and track data change paths, improve the transparency and interpretability of analysis logic, enable the system to execute complex reasoning processes in a certain logical order, so as to accurately identify data flow paths and their impacts.

[0045] Further, in the following, the present invention builds a Chain of Thought (CoT) system based on a large model using LangChain to implement five types of lineage analysis scenarios. After a user initiates a question, the large model first judges which of the five lineage analysis scenarios (five questions) the question belongs to, and then enters the corresponding lineage analysis scenario to give a response after the determination is made.

[0046] 1. Target table data status detection Configure the large intent recognition model. In a feasible embodiment, the specific prompt is as follows: Your task is intent recognition. Based on the content in {order}, understand the meaning of {order} and select the intent with the closest semantic meaning. The intents are: 1. Question about <data volume>; 2. Question about <daily flow details>; 3. Question about . Only answer 1, 2 or 3. Example 1: Recognize the sentence <The recent data volume of the dwd table is abnormal>, answer: 1. Example 2: Recognize the sentence <The recent flow of the dwd table is abnormal>, answer: 2. Example 3: Recognize the sentence <Check the daily flow of dwd>, answer: 2. Example 4: Recognize the sentence <Check the fields of the dwd table>, answer: 3. Recognize the intent of the sentence <{order}>. Your answer: .

[0047] Configure the large entity recognition model. In a feasible embodiment, the specific prompt is as follows: Your task is entity recognition. No matter what the question is, you need to identify the table name in the sentence <{content}> and answer in the format of: "What is the lineage of ?". Example 1: Identify the table name in the sentence <There is data missing in sw_outsummary for the last three days>. Answer: What is the lineage of sw_outsummary? Example 2: Identify the table name in the sentence <The sw_outsummary workflow is abnormal>. Answer: What is the lineage of sw_outsummary? Question: Identify the table name in the sentence <{content}>. Your answer: .

[0048] Configure a target table data status detection model (configure and set target model). In a feasible embodiment, the specific prompt words are as follows: Your task is to replace the contents of the string `select dt,count(*) as num from table where CAST(dt AS DATE)>= date_sub(current_date(), 3) group by dt order by dt desc;` according to the requirements. For example, replace the contents of `table` in `select dt,count(*) as num from table where CAST(dt AS DATE)>= date_sub(current_date(), 3) group by dt order by dt desc;` according to the following table structure: `<main table>:dwd.crm_memconsume, :ods.crm_memconsume,ods.crm_memconsume`. The answer is: To locate the abnormal data volume issue, I suggest referring to the following: --<Main table>: dwd.crm_memconsume selectdt,count(*) as num from dwd.crm_memconsume where CAST(dt AS DATE)>= date_sub(current_date(), 3) group by dt order by dt desc; --<Sub-table>: ods.crm_memconsume selectddt,count(*) as num from ods.crm_memconsume where CAST(dt AS DATE)>= date_sub(current_date(), 3) group by dt order by dt desc; --<Sub-table>: ods.crm_memconsume selectddt,count(*) as num from ods.crm_memconsume whereCAST(dt AS DATE)>= date_sub(current_date(), 3) group by dt order by Problem: Replace the table in the string `select dt,count(*) as num from table where CAST(dt ASDATE)>= date_sub(current_date(), 3) group by dt order by dt desc;` with the value {list}.The answer always starts with "For the abnormal positioning data volume issue, I suggest referring to:", is segmented by each note, and the SQL statements are displayed in separate lines. The answer is: .

[0049] By querying the data status of the target table, the SQL query statements for the target table and its bloodline sub-tables are returned, which facilitates the business to directly apply SQL to query the data status of the target table and its sub-tables. Through the above three steps of configuring the scenario thought chain, the final result is obtained.

[0050] 2. Daily transaction query of the target table Configure the intent recognition large model. In a feasible embodiment, the specific prompt is as follows: Your task is intent recognition. According to the content in {order}, understand the meaning of {order} and select the intent with the closest semantics. The intents are: 1. Questions about <data volume>; 2. Questions about <daily transaction details>; 3. Questions about ; Only answer 1 or 2 or 3. Example 1: Recognize the sentence <The recent data volume of table dwd is abnormal>, answer: 1. Example 2: Recognize the sentence <The recent transaction of table dwd is abnormal>, answer: 2. Example 3: Recognize the sentence <Check the daily transaction of dwd>, answer: 2. Example 4: Recognize the sentence <Check the fields of the dwd table>, answer: 3. Recognize the intent of the sentence <{order}>. Your answer: .

[0051] Configure the entity recognition large model. In a feasible embodiment, the specific prompt is as follows: Your task is entity recognition. No matter what the question is, you need to identify the table name in the sentence <{content}> and answer in the format of: "What is the bloodline of ?". Example 1: Identify the table name in the sentence <There is missing data in sw_outsummary in the past three days>. Answer: What is the bloodline of sw_outsummary? Example 2: Identify the table name in the sentence <The sw_outsummary workflow is abnormal>. Answer: What is the bloodline of sw_outsummary? Question: Identify the table name in the sentence <{content}>. Your answer: .

[0052] Configure the large model for daily transaction query of the target table (configure and set the target large model). In a feasible embodiment, the specific prompt is as follows: Your task is to replace the contents of the string `select dt,* from table where CAST(dt AS DATE)>= date_sub(current_date(), 3) order by dt desc;` according to the requirements. For example, replace the contents of `table` in `select dt,* from table where CAST(dt AS DATE)>= date_sub(current_date(),3) order by dt desc;` according to the following table structure: `<main table>: dwd.crm_memconsume, <sub-table>: ods.crm_memconsume, ods.crm_memconsume`. The answer is: To locate the abnormal transaction log issue, I suggest referring to the following: --<Main table>: dwd.crm_memconsume select dt,* from dwd.crm_memconsume where CAST(dt ASDATE)>= date_sub(current_date(), 3) order by dt desc; --<Sub-table>: ods.crm_memconsume select dt,* from ods.crm_memconsume where CAST(dt AS DATE)>= date_sub(current_date(), 3) order by dt desc; --<Sub-table>: ods.crm_memconsume selectdt,* from ods.crm_memconsume where CAST(dt AS DATE)>= date_sub(current_date(), 3) order by dt desc; Question: Replace the `table` element in the string `select dt,* from table where CAST(dtAS DATE)>= date_sub(current_date(), 3) order by dt desc;` with the format `{list}`. The answers always begin with "To locate abnormal transaction log issues, I suggest referring to:", are divided into sections based on each comment, and the SQL statement is displayed line by line. The answer is:

[0053] By analyzing the daily transaction data in the target table, a query statement for that table is returned, enabling quick business queries. Through the three steps of the above-described thought process for configuring this scenario, the final result is obtained.

[0054] 3. Generation of table creation statements Configure a large model for intent recognition. In a feasible embodiment, the specific prompt is as follows: Your task is intent recognition. Based on the content in {order}, understand the meaning of {order} and select the semantically closest one from the intents. The intents are: 1. Question about <data volume>; 2. Question about <daily transaction details>; 3. Question about . Only answer 1, 2 or 3. Example 1: Recognize the sentence <The recent data volume of the dwd table is abnormal>, answer: 1. Example 2: Recognize the sentence <The recent transaction of the dwd table is abnormal>, answer: 2. Example 3: Recognize the sentence <Check the daily transactions of dwd>, answer: 2. Example 4: Recognize the sentence <Check the table fields of dwd>, answer: 3. Recognize the intent of the sentence <{order}>. Your answer: .

[0055] Configure a large model for entity recognition. In a feasible embodiment, the specific prompt is as follows: Your task is entity recognition. No matter what the question is, you need to recognize the table name in the sentence <{content}> and answer in the format of: "What is the lineage of ?". Example 1: Recognize the table name in the sentence <There is missing data in sw_outsummary for the last three days>. Answer: What is the lineage of sw_outsummary? Example 2: Recognize the table name in the sentence <The sw_outsummary workflow is abnormal>. Answer: What is the lineage of sw_outsummary? Question: Recognize the table name in the sentence <{content}>. Your answer: .

[0056] Configure a large model for creating table statement generation (configure and set the target large model). In a feasible embodiment, the specific prompt is as follows: Your task is to replace the content in the string `show create table;` according to the requirements. For example, replace `table` in `show create table;` with: `<main table>:dwd.crm_memconsume,child table>:ods.crm_memconsume,ods.crm_memconsume`. The answer is: This is a query table field issue. I suggest referring to: `--<main table>:show create table dwd.crm_memconsume; --child table>:ods.crm_memconsume showcreate table ods.crm_memconsume; --child table>:ods.crm_memconsume ;`. The question is: Replace `table` in the string `show create table;` with `{list}`. The answer always begins with "Query table field issue, I suggest referring to:", breaks each comment into paragraphs, and displays the SQL statement on separate lines. The answer is:

[0057] By using the table creation statement for the target table in the question-and-answer process, the system returns the table creation statement for that table, enabling businesses to quickly perform table creation operations. Through the three steps of the above configuration process for this scenario, the final result is obtained.

[0058] 4. Formation of blood relations In a feasible embodiment, the configuration intent recognition model includes the following specific prompts: Your task is intent identification. Based on the content of `{order}`, understand the meaning of `{order}` and select the intent that is semantically closest. Intents include: 1. A question about <data volume>; 2. A question about <daily transaction details>; 3. A question about . Only answer 1, 2, or 3. Example 1: Identify the sentence <Recent data volume anomaly in table dwd>, answer: 1. Example 2: Identify the sentence <Recent transaction details anomaly in table dwd>, answer: 2. Example 3: Identify the sentence <View daily transaction details in dwd>, answer: 2. Example 4: Identify the sentence <View table fields in dwd>, answer: 3. Identify the intent of the sentence `{order}`. Your answer:

[0059] In a feasible embodiment, the specific prompt words for configuring a large entity recognition model are as follows: Your task is entity recognition. Regardless of the question, you need to identify the table name in the sentence <{content}> and answer in the format: "What is the lineage of ?". Example 1: Identify the table name in the sentence <sw_outsummary has data missing in the past three days>. Answer: What is the lineage of sw_outsummary? Example 2: Identify the table name in the sentence <sw_outsummary workflow is abnormal>. Answer: What is the lineage of sw_outsummary? Question: Identify the table name in the sentence <{content}>. Your answer: .

[0060] Configure the large model for lineage relationship generation (configure and set the target large model). In a feasible embodiment, the specific prompt words are as follows: Your task is to output the table lineage relationship according to the lineage list. For example: if the lineage list is "<Main table>: dwd.crm_memconsume, : ods.crm_memconsume, ods.crm_memconsume", then the answer is: "For the table lineage query, I suggest referring to the knowledge: the sub-tables of the main table dwd.crm_memconsume are: ods.crm_memconsume, ods.crm_memconsume." The answer always starts with "For the table lineage query, I suggest referring to:". Question: {list}. The answer is: .

[0061] This scenario returns the lineage sub-tables of the target table through question and answer about the lineage status of the target table, helping the business to quickly query the target. Through the above three steps of configuring the scenario's chain of thought, the final result is obtained.

[0062] 5. Other lineage related questions Configure the large model for intent recognition. In a feasible embodiment, the specific prompt words are as follows: Your task is intent recognition. Based on the content in {order}, understand the meaning of {order} and select the intent with the closest semantic meaning. The intents are: 1. Question about <data volume>; 2. Question about <daily flow details>; 3. Question about . Only answer 1, 2 or 3. Example 1: Identify the sentence <The recent data volume of table dwd is abnormal>, answer: 1. Example 2: Identify the sentence <The recent flow of table dwd is abnormal>, answer: 2. Example 3: Identify the sentence <Check the daily flow of dwd>, answer: 2. Example 4: Identify the sentence <Check the fields of the dwd table>, answer: 3. Identify the intent of the sentence <{order}>. Your answer: .

[0063] Configure the large model for entity recognition. In a feasible embodiment, the specific prompt words are as follows: Your task is entity recognition. Regardless of the question, you need to identify the table name in the sentence <{content}> and answer in the format of "What is the lineage of ?". Example 1: Identify the table name in the sentence <sw_outsummary has data missing in the past three days>. Answer: What is the lineage of sw_outsummary? Example 2: Identify the table name in the sentence <sw_outsummary workflow is abnormal>. Answer: What is the lineage of sw_outsummary? Question: Identify the table name in the sentence <{content}>. Your answer: .

[0064] Configure a large model for other lineage questions (configure and set the target large model). In a feasible embodiment, the specific prompt is as follows: Question: {list}. The answer is: .

[0065] By asking and answering lineage questions, the processing method for the question is returned, helping the business solve lineage problems quickly. Through the above three steps of configuring the scenario thought chain, the result is finally obtained.

[0066] Using the constructed large model Chain-of-Thought (CoT) system, the lineage analysis large model Agent can realize the localization of lineage problems and output suggestions. The model can receive questions or demands input by engineers, then distinguish the content of the question according to the query and answer the question.

[0067] Specifically, in an example, when an engineer inputs "Please query the lineage of oilcard.gas", the model may output "For the table lineage query, I suggest referring to the knowledge: the child tables of the main table oilcard.gas include: ods_oilcard.gas, ods_oilcard.gas, dwd_oilcard.tenantinfo." and provide corresponding suggestions.

[0068] Through data warehouse queries, the model can retrieve relevant knowledge, perform problem diagnosis, and provide answers.

[0069] The large model-based data lineage analysis method of the present invention can, based on large language model content generation technology, assist data warehouse developers to quickly locate data problems through lineage analysis question and answer, and improve the work efficiency of data application development. The data lineage analysis method of the present invention can not only help data warehouse personnel understand the data lineage, but also actively locate and analyze data problems, and provide specific solution suggestions.

[0070] Embodiment 2 As a specific embodiment of the present invention, this embodiment provides a large model-based data lineage analysis system, with reference to Figure 3It includes an SQL data collection and processing module 10, a vector knowledge base formation module 20, an interaction model agent construction module 30, and a lineage analysis and problem diagnosis module 40, wherein: The SQL data collection and processing module 10 is used to collect raw SQL data and process the collected raw SQL data to obtain standard SQL data. Vector knowledge base forming module 20 is used to form a vector knowledge base from the standard SQL data obtained after vectorization processing. The interaction model Agent construction module 30 is used to construct an interaction model Agent based on a large model according to the vector knowledge base; The lineage analysis and problem diagnosis module 40 is used to perform data lineage analysis on the vector knowledge base using the large model-based interaction model Agent, and generate data problem diagnosis results.

[0071] Specifically, in the interaction model Agent construction module 30, based on the base large language model, the vector knowledge base is embedded using langchain technology to construct a lineage analysis large model Agent that can interact with data warehouse personnel in natural language.

[0072] In the large-model-based data lineage analysis system of this invention, the SQL data collection and processing module 10 collects raw SQL data and standardizes it to convert it into a standard format for subsequent processing. The vector knowledge base formation module 20 vectorizes the standard SQL data, making it easier to apply artificial intelligence algorithms for analysis by converting SQL statements into vector representations. The resulting vector knowledge base is the foundation of the entire system, storing the vector representations of all processed SQL data. The interactive model agent construction module 30 builds an interactive model agent based on the previously formed vector knowledge base. The agent can understand complex query requests and provide corresponding responses based on the information in the vector knowledge base, which is crucial for achieving intelligent data management and querying. The lineage analysis and problem diagnosis module 40 uses the constructed interactive model agent to perform data lineage analysis on the vector knowledge base. By tracking and analyzing the data, it can not only understand the relationships between data but also identify potential problem points, thereby generating data problem diagnosis results to help users better understand and manage data assets.

[0073] The data lineage analysis system based on large models of the present invention is particularly suitable for enterprises or organizations that need to efficiently manage and analyze large amounts of data, and can provide strong support for data governance and other aspects.

[0074] Example 3 As a specific embodiment of the present invention, this embodiment provides a computer electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the data lineage analysis method based on a large model as described in Embodiment 1.

[0075] Example 4 As a specific embodiment of the present invention, this embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data lineage analysis method based on a large model as described in Embodiment 1.

[0076] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data lineage analysis method based on a large model as described in Embodiment 1.

[0077] The data lineage analysis method based on large models of the present invention can be widely applied in the fields of data warehousing and data application development services, thereby improving the efficiency of data application development work.

[0078] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A data lineage analysis method based on a large model, characterized in that, The method includes: Collect raw SQL data and process the collected raw SQL data to obtain standard SQL data; The standard SQL data obtained after vectorization processing forms a vector knowledge base; Based on the vector knowledge base, construct an interaction model Agent based on a large model; The large-model-based interaction model Agent is used to perform data lineage analysis on the vector knowledge base and generate data problem diagnosis results; in, The process of collecting raw SQL data and processing the collected raw SQL data to obtain standard SQL data includes: Collect table creation SQL data from the code repository to obtain the raw SQL data; The original SQL data is disassembled and extracted; Preprocess the decomposed and extracted SQL data; Extract information from the preprocessed SQL data; Save the extracted SQL data into a specific structure; Saving the extracted SQL data into a specific structure includes saving the extracted SQL data as {"lineage": target table, "sub-table structure": sub-table}; The standard SQL data obtained after vectorization processing forms a vector knowledge base by embedding the organized standard SQL data through the Data Connection component of langchain. The step of constructing an interaction model agent based on a large model according to the vector knowledge base includes using langchain to embed the vector knowledge base and constructing an interaction model agent based on a large model. The process of using the large-model-based interaction model Agent to perform data lineage analysis on the vector knowledge base and generate data problem diagnosis results includes constructing a CoT (Cooperative Knowledge Link) system based on the large model, configuring an intent recognition large model, configuring an entity recognition large model, and configuring a target setting large model. The intent recognition large model is used to understand the true purpose behind user queries or commands; the entity recognition large model is used to identify entities related to the data; and the target setting large model is used to clarify the specific purpose or problem that is desired to be solved through data lineage analysis. Using langchain, a CoT system based on a large model is built to realize five types of bloodline analysis scenarios. After a user raises a question, the large model first determines which of the five bloodline analysis scenarios it is, and then enters the corresponding bloodline analysis scenario to answer the question. By utilizing the well-constructed CoT (Cooperative Knowledge) system of the large-scale model, the bloodline analysis model Agent can locate bloodline issues and output suggestions. The model receives questions or requirements input by engineers, then identifies the content of the questions and answers them.

2. The data lineage analysis method based on a large model according to claim 1, characterized in that, The process of disassembling and extracting the original SQL data includes splitting multiple SQL segments into a single SQL statement during the disassembly process and extracting only the insert statement SQL by determining whether the SQL statement begins with the insert statement.

3. The data lineage analysis method based on a large model according to claim 2, characterized in that, The preprocessing of the disassembled and extracted SQL data includes: Delete the comments and remove all characters after the "--" symbol; Replace newline character , to remove all newline characters Replace " with a space; Replace tab character "\t" with a space; Remove extra spaces and replace consecutive spaces with a single space.

4. The data lineage analysis method based on a large model according to claim 3, characterized in that, The information extraction process for the preprocessed SQL data includes: extracting all table names, deleting duplicate table names, and removing the first table name after the insert_table statement and saving it as a target table, while saving the others as sub-tables.

5. A data lineage analysis system based on a large model, characterized in that, It includes modules for SQL data collection and processing, vector knowledge base formation, interaction model agent construction, and lineage analysis and problem diagnosis, among which: The SQL data collection and processing module is used to collect raw SQL data and process the collected raw SQL data to obtain standard SQL data. The vector knowledge base forming module is used to form a vector knowledge base from the standard SQL data obtained after vectorization processing. The interaction model Agent construction module is used to construct an interaction model Agent based on a large model according to the vector knowledge base; The lineage analysis and problem diagnosis module is used to perform data lineage analysis on the vector knowledge base using the large model-based interactive model Agent, and generate data problem diagnosis results. in, The process of collecting raw SQL data and processing the collected raw SQL data to obtain standard SQL data includes: Collect table creation SQL data from the code repository to obtain the raw SQL data; The original SQL data is disassembled and extracted; Preprocess the decomposed and extracted SQL data; Extract information from the preprocessed SQL data; Save the extracted SQL data into a specific structure; Saving the extracted SQL data into a specific structure includes saving the extracted SQL data as {"lineage": target table, "sub-table structure": sub-table}; The standard SQL data obtained after vectorization processing forms a vector knowledge base by embedding the organized standard SQL data through the Data Connection component of langchain. The step of constructing an interaction model agent based on a large model according to the vector knowledge base includes using langchain to embed the vector knowledge base and constructing an interaction model agent based on a large model. The process of using the large-model-based interaction model Agent to perform data lineage analysis on the vector knowledge base and generate data problem diagnosis results includes constructing a CoT (Cooperative Knowledge Link) system based on the large model, configuring an intent recognition large model, configuring an entity recognition large model, and configuring a target setting large model. The intent recognition large model is used to understand the true purpose behind user queries or commands; the entity recognition large model is used to identify entities related to the data; and the target setting large model is used to clarify the specific purpose or problem that is desired to be solved through data lineage analysis. Using langchain, a CoT system based on a large model is built to realize five types of bloodline analysis scenarios. After a user raises a question, the large model first determines which of the five bloodline analysis scenarios it is, and then enters the corresponding bloodline analysis scenario to answer the question. By utilizing the well-constructed CoT (Cooperative Knowledge) system of the large-scale model, the bloodline analysis model Agent can locate bloodline issues and output suggestions. The model receives questions or requirements input by engineers, then identifies the content of the questions and answers them.

6. A computer electronic device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-4.

Images