Data blood relationship analysis method, system and equipment based on large model and storage medium

Abstract

The invention provides a data consanguinity analysis method, system and equipment based on a large model and a storage medium, and belongs to the technical field of big data analysis application, and the method comprises the following steps: collecting original SQL data, and processing the collected original SQL data to obtain standard SQL data; the standard SQL data obtained after vectorization processing forms a vector knowledge base; according to the vector knowledge base, constructing an interaction model Agent based on a large model; and performing data consanguinity analysis on the vector knowledge base by using the interaction model Agent based on the large model, and generating a data problem diagnosis result. According to the data consanguinity analysis method, the data warehouse personnel can be helped to understand the consanguinity of the data, the data problem can be actively positioned and analyzed, specific solution suggestions are given, and the data application and development working efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data analysis application technology, and in particular to a data lineage analysis method, system, device and storage medium based on a large model. Background Art

[0002] Data lineage refers to a naturally formed relationship between data, similar to human lineage, throughout the entire life cycle of data, from its generation, processing, processing, fusion, flow to its final extinction. It not only involves the physical flow of data, but also includes the logical relationships and conversion processes of data. Data lineage analysis is a core function of metadata management and data governance tools. By establishing lineage relationships between data, it is possible to analyze whether changes in upstream data affect downstream related data; if the relationship between technical metadata and business metadata is established in metadata, lineage can be used to analyze the data flow between different business products and the business associations between different products; by analyzing data lineage, data can be better understood and used. For example, CN118467175A discloses a data lineage analysis method.

[0003] However, current data lineage analysis can only simply list the upstream and downstream relationships between data, that is, the associations between tables, and cannot answer questions related to lineage or provide solutions. Summary of the Invention

[0004] To address the above issues, the present invention provides a data lineage analysis method, system, device, and storage medium based on a large model, which helps data warehouse developers quickly locate data problems through lineage analysis questions and answers, thereby improving the efficiency of data application development.

[0005] The present invention provides a data lineage analysis method based on a large model, comprising: Collecting original SQL data and processing the collected original SQL data to obtain standard SQL data; The standard SQL data obtained after vectorization processing forms a vector knowledge base; According to the vector knowledge base, construct an interactive model Agent based on the large model; The interactive model Agent based on the large model is used to perform data lineage analysis on the vector knowledge base and generate data problem diagnosis results.

[0006] As a further improvement of the present invention, the collecting of raw SQL data and processing the collected raw SQL data to obtain standard SQL data includes: Collect SQL data and obtain original SQL data; Decomposing and extracting the original SQL data; Preprocess the SQL data after disassembly and extraction; Extract information from pre-processed SQL data; Save the extracted SQL data into a specific structure.

[0007] As a further improvement of the present invention, the decomposing and extracting the original SQL data includes decomposing multiple SQL segments into a single SQL and extracting only the insert statement SQL.

[0008] As a further improvement of the present invention, the pre-processing of the SQL data after disassembly and extraction includes: deleting the comment information, replacing the line break character " ", replace tab characters " \t ", and remove extra spaces.

[0009] As a further improvement of the present invention, the information extraction of the preprocessed SQL data includes: extracting all table names, deleting duplicate table names, and removing the first table name after the insert_table statement and saving it separately as a target table, and saving the others as sub-tables.

[0010] As a further improvement of the present invention, saving the SQL data after information extraction into a specific structure includes saving the SQL data after information extraction into {"bloodline": target table, "subtable structure": subtable}.

[0011] As a further improvement of the present invention, constructing the interactive model Agent based on the large model according to the vector knowledge base includes using langchain to embed the vector knowledge base to construct the interactive model Agent based on the large model.

[0012] The present invention provides a data lineage analysis system based on a large model, comprising an SQL data collection and processing module, a vector knowledge base formation module, an interactive model agent construction module, and a lineage analysis and problem diagnosis module, wherein: The SQL data collection and processing module is used to collect raw SQL data and process the collected raw SQL data to obtain standard SQL data; A vector knowledge base forming module is used to form a vector knowledge base from the standard SQL data obtained after vectorization processing; An interactive model Agent construction module is used to construct an interactive model Agent based on a large model according to the vector knowledge base; The lineage analysis and problem diagnosis module is used to perform data lineage analysis on the vector knowledge base using the interactive model Agent based on the large model and generate data problem diagnosis results.

[0013] The present invention provides a computer electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned large model-based data lineage analysis method.

[0014] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned large-model-based data lineage analysis method.

[0015] The present invention provides a data lineage analysis method, system, device and storage medium based on a large model. The method collects raw SQL data and processes the collected raw SQL data to obtain standard SQL data; the standard SQL data obtained after vectorization processing forms a vector knowledge base; based on the vector knowledge base, an interactive model agent based on a large model is constructed; the interactive model agent based on the large model is used to perform data lineage analysis on the vector knowledge base and generate data problem diagnosis results. Based on the large language model content generation technology, it can help data warehouse developers quickly locate data problems through lineage analysis questions and answers, thereby improving the efficiency of data application development. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a data lineage analysis method based on a large model according to an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of a specific analysis process of a data lineage analysis method based on a large model according to an embodiment of the present invention.

[0018] Figure 3 It is a structural diagram of data lineage analysis based on a large model in an embodiment of the present invention.

[0019] Explanation of reference numerals: 10. SQL data collection and processing module; 20. Vector knowledge base formation module; 30. Interaction model Agent construction module; 40. Blood relationship analysis and problem diagnosis module. DETAILED DESCRIPTION

[0020] The following is a combination of specific embodiments and appendix Figure 1-3 The invention is described in detail so that those skilled in the art can more fully understand the purpose, features and effects of the invention.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which the invention belongs. In the event that the definition of a term in the present invention conflicts with the meaning commonly understood by those skilled in the art to which the invention belongs, the definition in the present invention shall prevail.

[0022] In today's big data environment, data is experiencing explosive growth. To effectively manage big data and unlock its value, ensuring data quality and improving its credibility are crucial. Data lineage analysis can ensure this process is successful. By analyzing the lineage relationships between data, data lineage analysis clearly demonstrates the entire transformation process from its source to its current form. This makes data traceable and verifiable, significantly improving data quality and addressing issues such as data uncertainty and unreliability. Data lineage analysis allows users to clearly visualize the processing of data entities, facilitating data verification. Data lineage links intuitively demonstrate data lineage relationships.

[0023] The present invention improves existing data lineage analysis by providing a data lineage analysis method, system, device and storage medium based on a large model, so that data warehouse developers can quickly locate data problems.

[0024] Example 1 As a specific embodiment of the present invention, this embodiment provides a data lineage analysis method based on a large model, referring to Figure 1 、 Figure 2 , the specific steps are as follows: S100, collecting original SQL data, and processing the collected original SQL data to obtain standard SQL data; S200, forming a vector knowledge base from the standard SQL data obtained after vectorization processing; S300, constructing an interactive model Agent based on a large model according to the vector knowledge base; S400: Utilize the interactive model Agent based on the large model to perform data lineage analysis on the vector knowledge base and generate data problem diagnosis results.

[0025] The large-model-based data lineage analysis method provided by the present invention is based on large language model technology and can help data warehouse engineers quickly analyze data lineage problems.

[0026] Specifically, S100 collects raw SQL data and processes the collected raw SQL data to obtain standard SQL data including: S110, collect SQL data and obtain original SQL data SQL data includes table and field names and relationships between tables, which is essential information for lineage analysis. The original SQL data is obtained by collecting table creation SQL data from the code base.

[0027] S120: Decomposing and extracting the original SQL data When breaking down the raw SQL data, multiple SQL statements are broken down into a single SQL statement. A single SQL statement is easier to pinpoint than multiple SQL statements. This single SQL statement is used for subsequent content extraction. When multiple SQL statements are mixed together, the program cannot identify table and field information for regular expression extraction. Therefore, breaking down multiple SQL statements into a single SQL statement is necessary.

[0028] When extracting SQL data, only insert statements are extracted. Preferably, the SQL is extracted by determining whether the beginning of the SQL is insert.

[0029] Through the above processing, the obtained SQL can be made more accurate, the data capacity can be reduced, and unnecessary data can be reduced.

[0030] S130: Preprocess the SQL data after disassembly and extraction, including: Delete the remark information. Specifically, remove all characters after the "--" symbol. Replace newline character " Specifically, all newline characters " " is replaced with a space; Replace the tab character " \t ". Specifically, replace the tab character " \t " with a space. Delete extra spaces. Specifically, replace consecutive spaces with a single space.

[0031] Through the above processing, useless data carried in the original SQL data is eliminated, data redundancy is reduced, and the quality of SQL data is improved.

[0032] S140: extracting information from the pre-processed SQL data, including: Extract all table names; Delete duplicate table names; Remove the first table name after the insert_table statement and save it separately as the "target table", and the others as sub-tables.

[0033] By removing duplicates, the uniqueness of table names is maintained to avoid errors in subsequent data processing.

[0034] S150: Arrange the target table and sub-tables saved in S140 into a specific structure. Specifically, the specific structure is: {"lineage": target table, "sub-table structure": sub-table}.

[0035] By saving the target tables and sub-tables in the processed SQL data as a specific structure, it can better meet the standards of later processing, which helps improve the efficiency of later data use and facilitates the analysis of data lineage issues.

[0036] By processing the collected raw SQL data, removing junk data, extracting key information, and forming a data structure that can be used later, standard SQL data is obtained.

[0037] The data collection and processing step is used to collect SQL data, process it, output the table field relationship, and form the standard SQL data required for building the vector database (vector knowledge base).

[0038] The standard SQL data obtained after vectorization processing in step S200 is used to form a vector knowledge base, which includes embedding the sorted standard SQL data through the Data Connection component of Langchain to form the vector knowledge base. When questions are subsequently asked, the corresponding knowledge in the vector knowledge base is retrieved and fed into the large model for sorting and interpretation to form the answer.

[0039] Langchain technology provides a powerful set of tools and interfaces that streamline the entire process, from raw data collection to the formation of a vector knowledge base. Its Data Connection component makes it easier to connect disparate data sources, automating steps like data cleansing and conversion, thereby accelerating data processing. By converting SQL data into vector form, Langchain enables better understanding and manipulation of this data in high-dimensional space. This vectorized representation not only supports more precise data query and retrieval but also allows the application of complex machine learning models for data analysis.

[0040] S300 constructs an interactive model Agent based on a large model according to the vector knowledge base, including: using langchain to embed the vector knowledge base and construct an interactive model Agent based on a large model.

[0041] By using langchain technology and embedding the vector knowledge base formed in S200, we build a large-scale model agent for lineage analysis that can interact with data warehouse personnel in natural language, providing users with a more natural and efficient way to interact with data.

[0042] S400 utilizes the interactive model agent based on the large model to perform data lineage analysis on the vector knowledge base and generate data problem diagnosis results, including building a CoT system based on the large model, configuring the intent recognition large model, configuring the entity recognition large model, and configuring the target setting large model. By building the CoT system based on the large model, analytical capabilities and accuracy can be enhanced.

[0043] The intention recognition large model is used to understand the true purpose behind the user's query or command; the entity recognition large model is used to identify entities related to the data, such as table names; the goal setting large model is used to clarify the specific purpose or problem to be solved through data lineage analysis.

[0044] The Chain of Thought (CoT) is a method for solving problems through step-by-step reasoning. By constructing a CoT system, the interactive model Agent can more effectively understand and track the data change path, improve the transparency and interpretability of the analysis logic, enable the system to execute complex reasoning processes in a certain logical order, and thus accurately identify the data flow path and its impact.

[0045] Furthermore, hereinafter, the present invention uses langchain to construct a CoT system based on a large model to implement five types of data lineage analysis scenarios. After the user发起提问, the large model first determines which of the five data lineage analysis scenarios (five questions) it is, and then enters the corresponding data lineage analysis scenario to answer after confirmation. <见

[0046] 1. Detection of the situation of target table data Configure the intention recognition large model. In a feasible embodiment, the specific prompt words are as follows: Your task is intention recognition. According to the content in {order}, understand the meaning of {order} and select the one with the closest semantics in the intention. The intentions are: 1. Questions about <data volume>; 2. Questions about <daily transaction details>; 3. Questions about . Only answer 1 or 2 or 3. Example 1: Identify the sentence <The data volume of table dwd is abnormal recently>, answer: 1. Example 2: Identify the sentence <The workflow of table dwd is abnormal>, answer: 2. Example 3: Identify the sentence <View the daily transactions of dwd>, answer: 2. Example 4: Identify the sentence <View the table fields of dwd>, answer: 3. Identify the intention of the sentence <{order}>. Your answer:.

[0047] Configure the entity recognition large model. In a feasible embodiment, the specific prompt words are as follows: Your task is entity recognition. Regardless of the question, you need to identify the table name in the sentence <{content}> and answer in the format: "What is the lineage of ?". Example 1: Identify the table name in the sentence <There is data missing in sw_outsummary in the past three days>. Answer: What is the lineage of sw_outsummary? Example 2: Identify the table name in the sentence <The workflow of sw_outsummary is abnormal>. Answer: What is the lineage of sw_outsummary? Question: Identify the table name in the sentence <{content}>. Your answer:.

[0048] Configure the target table data status detection model (configure the target setting model). In a feasible embodiment, the specific prompt words are as follows: Your task is to replace the contents of the string "select dt, count(*) as num from table where CAST(dt AS DATE)>= date_sub(current_date(), 3) group by dt order by dt desc;" as required. For example, replace "table" in "select dt, count(*) as num from table where CAST(dt ASDATE)>= date_sub(current_date(), 3) group by dt order by dt desc;" with "<main table>:dwd.crm_memconsume,:ods.crm_memconsume,ods.crm_memconsume." The answer is: To locate the problem of abnormal data volume, I suggest referring to: --<main table>:dwd.crm_memconsume select dt,count(*) as num from dwd.crm_memconsume where CAST(dt AS DATE)>= date_sub(current_date(), 3) group by dt order by dt desc; --:ods.crm_memconsume select dt,count(*) as num from ods.crm_memconsume where CAST(dt AS DATE)>= date_sub(current_date(), 3) group by dt order by dt desc; --:ods.crm_memconsume select dt,count(*) as num from ods.crm_memconsume where CAST(dt AS DATE)>= date_sub(current_date(), 3) group by dt order by dtdesc;. Question: Replace table in the string select dt,count(*) as num from table where CAST(dt ASDATE)>= date_sub(current_date(), 3) group by dt order by dt desc; with {list}.The answer always starts with "Locate the abnormal data volume problem. I suggest referring to:", and is segmented according to each note, and the SQL statements are shown in separate lines. The answer is:.

[0049] By querying the data situation of the target table in the question and answer, return the SQL query statements of the target table and its lineage sub-tables, so that the business can directly use the SQL to query the data situation of the target table and its sub-tables. Through the above three steps of the thinking chain of this scenario, the final result is obtained.

[0050] 2. Query the daily transactions of the target table Configure the large model for intent recognition. In a feasible embodiment, the specific prompt words are as follows: Your task is intent recognition. According to the content in {order}, understand the meaning of {order} and select the one with the closest semantics in the intent. The intents are: 1. Questions about <data volume>; 2. Questions about <daily transaction details>; 3. Questions about ; Only answer 1 or 2 or 3. Example 1: Identify the sentence <The data volume of table dwd is abnormal recently>, answer: 1. Example 2: Identify the sentence <The workflow of table dwd is abnormal>, answer: 1. Example 3: Identify the sentence <View the daily transactions of dwd>, answer: 2. Example 4: Identify the sentence <View the fields of table dwd>, answer: 3. Identify the intent of the sentence <{order}>. Your answer:.

[0051] Configure the large model for entity recognition. In a feasible embodiment, the specific prompt words are as follows: Your task is entity recognition. No matter what the question is, you need to identify the table name in the sentence <{content}> and answer in the format of: "What is the lineage of ?". Example 1: Identify the table name in the sentence <There is data missing in sw_outsummary in the past three days>. Answer: What is the lineage of sw_outsummary? Example 2: Identify the table name in the sentence <The workflow of sw_outsummary is abnormal>. Answer: What is the lineage of sw_outsummary? Question: Identify the table name in the sentence <{content}>. Your answer:.

[0052] Configure the large model for querying the daily transactions of the target table (configure the set target large model). In a feasible embodiment, the specific prompt words are as follows: Your task is to replace the contents of the string "select dt,* from table where CAST(dt ASDATE)>= date_sub(current_date(), 3) order by dt desc;" as required. For example, replace "table" in "select dt,* from table where CAST(dt AS DATE)>= date_sub(current_date(),3) order by dt desc;" with "<main table>:dwd.crm_memconsume,:ods.crm_memconsume,ods.crm_memconsume." The answer is: To locate the abnormal transaction flow problem, I suggest referring to: --<main table>:dwd.crm_memconsume select dt,* from dwd.crm_memconsume where CAST(dt ASDATE)>= date_sub(current_date(), 3) order by dt desc; --:ods.crm_memconsume select dt,* from ods.crm_memconsume where CAST(dt AS DATE)>= date_sub(current_date(), 3) order by dt desc; --:ods.crm_memconsume selectdt,* from ods.crm_memconsume where CAST(dt AS DATE)>= date_sub(current_date(), 3) order by dt desc;. Question: Replace the table in the string "select dt,* from table where CAST(dtAS DATE)>= date_sub(current_date(), 3) order by dt desc;" with {list}. The answer always begins with "To troubleshoot abnormal bank flow issues, I recommend referring to:" and then breaks down the SQL statements into sections according to each comment, presenting them in separate lines. The answer is:

[0053] By asking and answering about the daily transaction flow of the target table, we can return a query statement for the transaction flow in this table, helping businesses quickly query it. By configuring the three steps of the thought chain for this scenario, we ultimately reach the result.

[0054] 3. Generate table creation statements Configure the intent recognition large model. In a feasible embodiment, the specific prompt words are as follows: Your task is intent recognition. According to the content in {order}, understand the meaning of {order} and select the one with the closest semantics from the intents. The intents are: 1. Questions about <data volume>; 2. Questions about <daily transaction details>; 3. Questions about . Only answer 1 or 2 or 3. Example 1: Identify the sentence <The data volume of table dwd is abnormal recently>, answer: 1. Example 2: Identify the sentence <The transaction of table dwd is abnormal recently>, answer: 2. Example 3: Identify the sentence <View the daily transactions of dwd>, answer: 2. Example 4: Identify the sentence <View the table fields of dwd>, answer: 3. Identify the intent of the sentence <{order}>. Your answer:.

[0055] Configure the entity recognition large model. In a feasible embodiment, the specific prompt words are as follows: Your task is entity recognition. Regardless of the question, you need to identify the table name in the sentence <{content}> and answer in the format: "What is the lineage of ?" Example 1: Identify the table name in the sentence <There is data missing in sw_outsummary in the past three days>. Answer: What is the lineage of sw_outsummary? Example 2: Identify the table name in the sentence <The workflow of sw_outsummary is abnormal>. Answer: What is the lineage of sw_outsummary? Question: Identify the table name in the sentence <{content}>. Your answer:.

[0056] Configure the table creation statement generation large model (configure the set target large model). In a feasible embodiment, the specific prompt words are as follows: Your task is to replace the contents of the string show create table; according to the requirements. For example: replace the table in show create table; with: <main table>:dwd.crm_memconsume,:ods.crm_memconsume,ods.crm_memconsume. The answer is: Regarding querying table fields, I suggest referring to: --<main table>:show create table dwd.crm_memconsume; --:ods.crm_memconsume showcreate table ods.crm_memconsume; --:ods.crm_memconsume;. Question: Replace the table in the string show create table; with {list}. The answer always starts with "Questions about querying table fields, I suggest referring to:", divides into paragraphs according to each comment, and displays the SQL statement in separate lines. The answer is:.

[0057] By answering the query and answering the target table creation statement, the table creation statement is returned, helping businesses quickly create the table. By configuring the three steps of the scenario thinking chain above, the final result is obtained.

[0058] 4. Generation of blood relationship Configure the intent recognition model. In a feasible embodiment, the specific prompt words are as follows: Your task is to identify intents. Based on the content of {order}, understand its meaning and select the one with the closest semantics from the intents. The intents include: 1. Questions about <data volume>; 2. Questions about <daily transaction details>; 3. Questions about . Answer only 1, 2, or 3. Example 1: Identify the sentence <Abnormal recent data volume in table DWD>. Answer: 1. Example 2: Identify the sentence <Abnormal recent transaction volume in table DWD>. Answer: 2. Example 3: Identify the sentence <Check DWD daily transaction>. Answer: 2. Example 4: Identify the sentence <Check DWD table fields>. Answer: 3. Identify the intent of the sentence <{order}>. Your answer: .

[0059] Configuring the entity recognition model, in a feasible embodiment, the specific prompt words are as follows: Your task is entity recognition. Regardless of the question, you need to identify the table name in the sentence <{content}> and answer in the format: "What is the lineage of ?" Example 1: Identify the table name in the sentence <sw_outsummary has data missing in the last three days>. Answer: What is the lineage of sw_outsummary? Example 2: Identify the table name in the sentence <sw_outsummary workflow is abnormal>. Answer: What is the lineage of sw_outsummary? Question: Identify the table name in the sentence <{content}>. Your answer:.

[0060] Configure the lineage generation large model (configure the target large model). In a feasible embodiment, the specific prompt words are as follows: Your task is to output the table lineage relationship based on the lineage list. For example, if the lineage list is "<master table>: dwd.crm_memconsume, : ods.crm_memconsume, ods.crm_memconsume", then answer: "Regarding the query of table lineage, I suggest referring to the knowledge: The sub tables of the master table dwd.crm_memconsume are: ods.crm_memconsume, ods.crm_memconsume." The answer always starts with "Regarding the query of table lineage, I suggest referring to:". Question: {list}. The answer is:.

[0061] By asking about the lineage of the target table and returning the lineage sub tables of the table, it helps the business quickly query the target. Through the above three steps of configuring the thinking chain of this scenario, the final result is obtained.

[0062] 5. Other lineage questions Configure the intent recognition large model. In a feasible embodiment, the specific prompt words are as follows: Your task is intent recognition. Based on the content in {order}, understand the meaning of {order} and select the semantically closest one among the intents. The intents are: 1. Questions about <data volume>; 2. Questions about <daily transaction details>; 3. Questions about ; Only answer 1 or 2 or 3. Example 1: Identify the sentence <The data volume of table dwd is abnormal recently>, answer: 1. Example 2: Identify the sentence <The daily transactions of table dwd are abnormal recently>, answer: 2. Example 3: Identify the sentence <View the daily transactions of dwd>, answer: 2. Example 4: Identify the sentence <View the fields of table dwd>, answer: 3. Identify the intent of the sentence <{order}>. Your answer:.

[0063] Configure the entity recognition large model. In a feasible embodiment, the specific prompt words are as follows: Your task is entity recognition. Regardless of the question, you need to identify the table name in the sentence <{content}> and answer in the format: "What is the lineage of ?". Example 1: Identify the table name in the sentence <sw_outsummary has data missing in the last three days>. Answer: What is the lineage of sw_outsummary? Example 2: Identify the table name in the sentence <sw_outsummary workflow is abnormal>. Answer: What is the lineage of sw_outsummary? Question: Identify the table name in the sentence <{content}>. Your answer:.

[0064] Configure other lineage problem large models (configure the set target large model). In a feasible embodiment, the specific prompt words are as follows: Question: {list}. The answer is:.

[0065] By asking and answering lineage questions, return the processing method of the question to help the business quickly solve lineage problems. Through the above three steps of configuring the scenario thought chain, the final result is obtained.

[0066] Using the constructed large model thought chain CoT system, the lineage problem can be located and suggestions can be output according to the lineage analysis large model Agent. The model can receive the questions or requirements input by engineers, and then distinguish the question content according to the question and answer the question.

[0067] Specifically, in an example, when an engineer inputs "Please query the lineage of oilcard.gas?", the model can output "To query the table lineage problem, I suggest referring to the knowledge: The sub-tables of the main table oilcard.gas are: ods_oilcard.gas, ods_oilcard.gas, dwd_oilcard.tenantinfo." and give corresponding suggestions.

[0068] Through data warehouse questions, the model can retrieve relevant knowledge, diagnose problems, and answer them.

[0069] The data lineage analysis method based on large models of the present invention can help data warehouse developers quickly locate data problems through lineage analysis questions and answers based on the content generation technology of large language models, and improve the work efficiency of data application development. According to the data lineage analysis method of the present invention, it can not only help data warehouse personnel understand the lineage of data, but also actively locate and analyze data problems and give specific solutions.

[0070] Embodiment 2 As a specific embodiment of the present invention, this embodiment provides a data lineage analysis system based on large models. Refer to Figure 3, including SQL data collection and processing module 10, vector knowledge base formation module 20, interactive model agent construction module 30, blood relationship analysis and problem diagnosis module 40, wherein: The SQL data collection and processing module 10 is used to collect raw SQL data and process the collected raw SQL data to obtain standard SQL data; A vector knowledge base forming module 20 is configured to form a vector knowledge base from the standard SQL data obtained after vectorization processing; An interactive model Agent construction module 30 is used to construct an interactive model Agent based on a large model according to the vector knowledge base; The lineage analysis and problem diagnosis module 40 is used to perform data lineage analysis on the vector knowledge base using the interactive model Agent based on the large model and generate data problem diagnosis results.

[0071] Specifically, in the interactive model agent construction module 30, based on the base large language model, the langchain technology is used to embed the vector knowledge base to build a blood relationship analysis large model agent that can interact with data warehouse personnel in natural language.

[0072] In the data lineage analysis system based on a large model of the present invention, the SQL data collection and processing module 10 collects the original SQL data and standardizes the data to convert the original SQL data into a standard format for subsequent processing; the vector knowledge base formation module 20 vectorizes the standard SQL data. By converting SQL statements into vector representations, artificial intelligence algorithms can be more conveniently applied for analysis. The formed vector knowledge base is the basis of the entire system and stores the vector representation of all processed SQL data; the interactive model agent construction module 30 constructs an interactive model agent based on the vector knowledge base formed above. The agent can understand complex query requests and provide corresponding responses based on the information in the vector knowledge base, which is crucial for realizing intelligent data management and query; the lineage analysis and problem diagnosis module 40 uses the constructed interactive model agent to perform data lineage analysis on the vector knowledge base. By tracking and analyzing the data, it can not only understand the relationship between the data, but also identify potential problem points, thereby generating data problem diagnosis results to help users better understand and manage data assets.

[0073] The large-model-based data lineage analysis system of the present invention is particularly suitable for enterprises or organizations that need to efficiently manage and analyze large amounts of data, and can provide strong support for data governance and other aspects.

[0074] Example 3 As a specific embodiment of the present invention, this embodiment provides a computer electronic device, including a memory, a processor and a computer program stored on the memory, and the processor executes the computer program to implement the steps of the large model-based data lineage analysis method described in Example 1.

[0075] Example 4 As a specific embodiment of the present invention, this embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the data lineage analysis method based on the large model described in Example 1 are implemented.

[0076] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the large model-based data lineage analysis method described in Example 1.

[0077] The large-model-based data lineage analysis method of the present invention can be widely used in the fields of data warehouses and data application development services to improve the efficiency of data application development work.

[0078] The above description is merely a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent variation based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A data lineage analysis method based on a large model, characterized in that: The method comprises: Collecting original SQL data and processing the collected original SQL data to obtain standard SQL data; The standard SQL data obtained after vectorization processing forms a vector knowledge base; According to the vector knowledge base, construct an interactive model Agent based on the large model; The interactive model Agent based on the large model is used to perform data lineage analysis on the vector knowledge base and generate data problem diagnosis results.

2. The data lineage analysis method based on a large model according to claim 1 is characterized in that: The collecting of raw SQL data and processing the collected raw SQL data to obtain standard SQL data includes: Collect SQL data and obtain original SQL data; Decomposing and extracting the original SQL data; Preprocess the SQL data after disassembly and extraction; Extract information from pre-processed SQL data; Save the extracted SQL data into a specific structure.

3. The data lineage analysis method based on a large model according to claim 1 is characterized in that: The decomposing and extracting the original SQL data includes decomposing multiple SQL segments into a single SQL segment and extracting only the insert statement SQL segment.

4. The data lineage analysis method based on a large model according to claim 3 is characterized in that: The pre-processing of the SQL data after disassembly and extraction includes: deleting comment information, replacing line breaks, ", replace tab characters " \t ", and remove extra spaces.

5. The data lineage analysis method based on a large model according to claim 4 is characterized in that: The information extraction of the pre-processed SQL data includes: extracting all table names, deleting duplicate table names, and removing the first table name after the insert_table statement and saving it separately as a target table, and saving the others as sub-tables.

6. The data lineage analysis method based on a large model according to claim 5 is characterized in that: Saving the SQL data after information extraction into a specific structure includes saving the SQL data after information extraction into {"lineage": target table, "sub-table structure": sub-table}.

7. The data lineage analysis method based on a large model according to claim 1 is characterized in that: The step of constructing the interactive model Agent based on the large model according to the vector knowledge base includes using langchain to embed the vector knowledge base and construct the interactive model Agent based on the large model.

8. A data lineage analysis system based on a large model, characterized by: It includes SQL data collection and processing module, vector knowledge base formation module, interactive model agent construction module, and lineage analysis and problem diagnosis module, among which: The SQL data collection and processing module is used to collect raw SQL data and process the collected raw SQL data to obtain standard SQL data; A vector knowledge base forming module is used to form a vector knowledge base from the standard SQL data obtained after vectorization processing; An interactive model Agent construction module is used to construct an interactive model Agent based on a large model according to the vector knowledge base; The lineage analysis and problem diagnosis module is used to perform data lineage analysis on the vector knowledge base using the interactive model Agent based on the large model and generate data problem diagnosis results.

9. A computer electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images