Data consanguinity acquisition method and device based on large language model, storage medium and processor

Abstract

The invention discloses a data consanguinity acquisition method and device based on a large language model, a storage medium and a processor. According to the scheme, firstly, an SQL statement to be subjected to blood relationship analysis is obtained, and the SQL statement serves as input of a large language model; extracting metadata of a table involved in the SQL statement by utilizing the large language model; and finally, the large language model outputs a blood relationship analysis result containing the SQL statement according to the SQL statement and the metadata of the table involved in the SQL statement and the constructed cue word. Compared with the prior art that the data blood relationship information is obtained by manually analyzing SQL statements, the method has the advantages that the accuracy of data blood relationship analysis is improved by means of a large language model, professional knowledge of database experts is not relied on any more, a data blood relationship analysis function is achieved without writing complex codes, and the overall input cost is reduced. Compared with an analysis scheme in the prior art, the method has obvious advantages.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a method, device, storage medium and processor for acquiring data lineage based on a large language model. Background Art

[0002] Data lineage plays a vital role in the field of big data. It refers to the ability to track the source, conversion process, and final use of each table or even each field in the data warehouse. This ability is essential to ensure the transparency, traceability, and accuracy of data. Data lineage is an indispensable component in the field of big data. It plays an irreplaceable role in improving the credibility of data, ensuring compliance, optimizing data governance, and supporting advanced data analysis technologies.

[0003] The existing data lineage acquisition method is mainly achieved through manual analysis of SQL statements. Manual SQL statement analysis has shortcomings in accuracy and operability. For example, when the SQL statement is relatively complex, it is difficult to parse such complex SQL statements. If the person parsing the SQL statement is not professional enough or makes mistakes, the accuracy of the parsed data lineage is affected.

[0004] In addition, SQL syntax has standards such as SQL-2016 and SQL-2019 under the ISO standard. Different database systems (such as Oracle, MySQL, PostgreSQL, Microsoft SQL Server, etc.) may have their own specific SQL dialects and extensions. These dialects may not fully comply with standard SQL, and each will add some platform-specific features and optimizations. This means that even for the same type of database operations, different manufacturers may have completely different writing methods.

[0005] SQL statement parsing has a very high technical threshold in development and maintenance. The R&D team must first be familiar with the syntax standards, SQL dialects, and proprietary functional extensions of various database systems, and then abstract the core functional points on this basis, and finally implement the functional points through programming languages. The entire process requires the cooperation of multiple roles and multiple technologies, and the threshold and investment cost are very high, which is not conducive to the widespread popularization of this parsing technology. Summary of the invention

[0006] Based on the above problems, the present application provides a method, device, storage medium and processor for acquiring data lineage based on a large language model, with the aim of improving the accuracy of data lineage acquisition and reducing costs.

[0007] The embodiments of the present application disclose the following technical solutions:

[0008] In a first aspect, the present application provides a method for acquiring data lineage based on a large language model, the method comprising:

[0009] Obtaining a SQL statement to be analyzed by lineage analysis, and using the SQL statement as input of a large language model;

[0010] The large language model is used to extract metadata of the table involved in the SQL statement; wherein the metadata of the table includes a table creation statement; the table creation statement includes a table name, a table remark, a field name, a field type, and a field remark;

[0011] The large language model outputs a lineage analysis result including the SQL statement according to the constructed prompt words based on the SQL statement and the metadata of the table involved in the SQL statement; wherein the constructed prompt words include key content involved in the lineage analysis result required to be output, so as to guide the large language model to generate an expected response process; the lineage analysis result of the SQL statement includes the source and transformation process of each field in the SQL statement.

[0012] Before obtaining the SQL statement to be analyzed by blood relationship, it also includes:

[0013] A large language model with an actual score greater than a preset score is selected through comprehensive evaluation, sample data is prepared, and the selected large language model is fine-tuned using the sample data.

[0014] The step of selecting a large language model whose actual score value is greater than a preset score value through comprehensive evaluation includes:

[0015] An evaluation data set is constructed using SQL statements that are actually running online; the evaluation data set includes the SQL statements and the source table, source field, target table, target field, and processing method of the SQL statements; the SQL statements cover multiple types of writing scenarios in terms of SQL type distribution, and cover multiple levels of complexity in terms of SQL statement complexity;

[0016] Based on the evaluation data set, the existing large language model is trained, and the performance parameters and related capability indicators of the existing large language model are weighted according to the task requirements of lineage analysis, and the actual score value of the existing large language model is calculated, and the large language model with an actual score value greater than the preset score value is selected as the large language model for data lineage acquisition; the performance parameters of the existing large language model include accuracy and recall rate; the related capability indicators include complexity handling capability, robustness and error analysis, time series analysis and interpretability.

[0017] The preparing sample data comprises:

[0018] Collect and organize SQL statement samples corresponding to the database system; the SQL statement samples include SQL syntax descriptions and examples, function usage instructions and examples, and SQL statements that have been executed in the database; the SQL syntax descriptions and examples shall be based on those published on the official website of each database system; the SQL statements that have been executed in the database shall be based on those that have been successfully executed.

[0019] The fine-tuning of the selected large language model using the sample data includes:

[0020] Removing duplicate and invalid samples from the sample data, marking the relationships between tables and fields in the SQL statements contained in the sample data, and dividing the data set consisting of the sample data into a training set, a validation set, and a test set;

[0021] Select a deep learning framework, and install the library corresponding to the selected deep learning framework and the selected deep learning framework;

[0022] Loading the selected large language model;

[0023] Defining tasks to be performed by the selected large language model, and fine-tuning the selected large language model using the training set; the tasks include SQL parsing and data lineage analysis; the fine-tuning achieves the best effect of the selected large language model by adjusting hyperparameters; the hyperparameters include learning rate and batch size;

[0024] Verifying the selected large language model after fine-tuning on the validation set and testing it on the test set;

[0025] The selected large language model that has been verified and tested is saved.

[0026] The construction method of the prompt word includes:

[0027] Describe the specific context in which the task occurs in the large language model;

[0028] Defining a task objective to be performed by the large language model;

[0029] indicating a writing style of the large language model;

[0030] Defining the attitude and emotional color of the large language model response;

[0031] Defining a specific group that receives responses from the large language model;

[0032] The structural form of the large language model response is preset.

[0033] The second aspect of the present application provides a data lineage acquisition device based on a large language model, the device comprising:

[0034] A lineage analysis statement acquisition module is used to acquire SQL statements to be analyzed by lineage, and use the SQL statements as input of the large language model;

[0035] A metadata extraction module, used to extract metadata of the table involved in the SQL statement using the large language model; wherein the metadata of the table includes a table creation statement; the table creation statement includes a table name, a table remark, a field name, a field type, and a field remark;

[0036] A lineage analysis result output module is used for the large language model to output the lineage analysis result containing the SQL statement according to the constructed prompt words based on the SQL statement and the metadata of the table involved in the SQL statement; wherein the constructed prompt words include key content involved in the lineage analysis result required to be output, so as to guide the large language model to generate an expected response process; the lineage analysis result of the SQL statement includes the source and transformation process of each field in the SQL statement.

[0037] A data lineage acquisition device based on a large language model also includes:

[0038] Large language model selection module, sample data preparation module and large language model fine-tuning module.

[0039] A third aspect of the present application provides a computer-readable storage medium, in which a computer program is stored. When the program is executed by a processor, a method for acquiring data lineage based on a large language model as provided in any implementation of the first aspect is implemented.

[0040] A fourth aspect of the present application provides a processor for running a computer program, which, when running, executes a data lineage acquisition method based on a large language model as provided in any implementation of the first aspect.

[0041] Compared with the prior art, this application has the following beneficial effects:

[0042] This application uses a large language model to learn SQL syntax, which can adapt to various complex SQL statements and various SQL dialects, and has greatly improved accuracy; the large language model anthropomorphically analyzes and answers the data lineage involved in SQL statements, and does not require database experts and complex code programming. Technical personnel only need to focus on collecting SQL syntax documents for various database systems, conducting model training and tuning, and writing more complete prompt word commands and other relatively low-threshold tasks, thereby reducing investment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A flow chart of a method for acquiring data lineage based on a large language model provided in an embodiment of the present application;

[0045] Figure 2 A flowchart of a method for acquiring data lineage based on a large language model provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the structure of a data lineage acquisition device based on a large language model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] As described above, the current method of obtaining data lineage is mainly through manual analysis of SQL statements. Manual analysis of SQL statements has shortcomings in accuracy and operability: SQL statements are relatively complex, and there are many standards for SQL syntax. Different databases have their own specific SQL dialects and extensions. Therefore, SQL statement analysis has a very high technical threshold in development and maintenance. R&D personnel must be familiar with the syntax standards, dialects, and proprietary functional extensions of various databases, and then abstract the core functional points on this basis. Finally, the functional points are implemented through programming languages. This process requires the cooperation of multiple roles and multiple technologies, and the implementation threshold and investment cost are very high.

[0048] In view of the above problems, the inventors have proposed a data lineage acquisition method, device, storage medium and processor based on a large language model after research. First, the SQL statement to be analyzed for lineage is obtained, and the SQL statement is used as the input of the large language model; then the large language model is used to extract the metadata of the table involved in the SQL statement; finally, the large language model outputs the lineage analysis result containing the SQL statement according to the constructed prompt words based on the SQL statement and the metadata of the table involved in the SQL statement. This solution uses a large language model to learn SQL syntax, allowing the large language model to anthropomorphically analyze and answer the data lineage involved in the SQL statement, so that the accuracy of data lineage analysis is improved, no longer relying on the professional knowledge of database experts, and no longer needing to write complex code to implement the data lineage analysis function, so that the overall investment cost can be reduced.

[0049] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] See also Figure 1 , which is a flow chart of a method for acquiring data lineage based on a large language model provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0051] S101. Obtain an SQL statement to be analyzed by blood relationship, and use the SQL statement as input of a large language model.

[0052] The SQL statement can generally be obtained and exported from a database system; the large language model usually refers to an artificial intelligence model with strong natural language processing capabilities, which can understand, parse and generate complex text content, such as qwen1p5-110B-chat-baseline, Xinghuo 65B-1024-sft, Xinghuo-coder-13B-baseline and deepseek-coder-33B.

[0053] S102: Extract metadata of the table involved in the SQL statement using the large language model.

[0054] The metadata of the table includes a table creation statement; the table creation statement includes a table name, a table remark, a field name, a field type, and a field remark.

[0055] The metadata of the tables involved are extracted from the SQL statements through a large language model. The metadata information can help understand the structure and purpose of the tables.

[0056] S103: The large language model outputs a lineage analysis result including the SQL statement according to the constructed prompt words based on the SQL statement and metadata of the table involved in the SQL statement.

[0057] Among them, the constructed prompt words include key content involved in the lineage analysis results required to be output, so as to guide the large language model to produce the expected response process; the lineage analysis results of the SQL statement include the source and transformation process of each field in the SQL statement.

[0058] The large language model is used to parse the metadata information of the table involved in the SQL statement, and the large language model is guided to generate a lineage analysis result through a pre-constructed prompt word. The prompt word is a guiding text for guiding the output of a specific type of the large language model. Here, the prompt word contains the key content involved in the lineage analysis result required to be output, so as to ensure that the model generates a lineage analysis result that meets the requirements.

[0059] The large language model can process complex SQL statements and adapt to different SQL dialects, thereby improving the accuracy of data lineage analysis. Using the large language model for data lineage analysis no longer relies on the expertise of database experts, nor does it require writing complex code to implement data lineage analysis functions, thereby reducing the overall investment cost.

[0060] The data lineage acquisition method based on the large language model introduced in the above embodiment also includes selecting a large language model whose actual score value is greater than a preset score value through comprehensive evaluation before obtaining the SQL statement to be analyzed by lineage, preparing sample data, and using the sample data to fine-tune the selected large language model. The following embodiment will be described in detail.

[0061] See also Figure 2 , which is a flow chart of another method for acquiring data lineage based on a large language model provided in an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0062] S201. Select a large language model whose actual score is greater than a preset score through comprehensive evaluation.

[0063] An evaluation data set is constructed using SQL statements that are actually running online; the evaluation data set includes the SQL statements and the source table, source field, target table, target field, and processing method of the SQL statements; the SQL statements cover multiple types of writing scenarios in terms of SQL type distribution, and cover multiple levels of complexity in terms of SQL statement complexity;

[0064] Based on the evaluation data set, the existing large language model is trained, and the performance parameters and related capability indicators of the existing large language model are weighted according to the task requirements of lineage analysis, and the actual score value of the existing large language model is calculated, and the large language model with an actual score value greater than the preset score value is selected as the large language model for data lineage acquisition; the performance parameters of the existing large language model include accuracy and recall rate; the related capability indicators include complexity handling capability, robustness and error analysis, time series analysis and interpretability.

[0065] S202: Prepare sample data.

[0066] Collect and organize SQL statement samples corresponding to the database system; the SQL statement samples include SQL syntax descriptions and examples, function usage instructions and examples, and SQL statements that have been executed in the database; the SQL syntax descriptions and examples shall be based on those published on the official website of each database system to avoid mixing in negative samples; the SQL statements that have been executed in the database shall be based on those that have been successfully executed.

[0067] Taking the instructions for using the SUBSTR function as an example, the command format is string substr(string <str>,bigint<start_position>[,bigint <length>]) returns a substring of the string str starting at start position and with a length of length, where start position and length are of BIGINT type.

[0068] S203: Use the sample data to fine-tune the selected large language model.

[0069] Remove duplicate and invalid samples from the sample data, mark the relationship between tables and fields in the SQL statements contained in the sample data, and divide the data set consisting of the sample data into a training set, a validation set, and a test set, with the usual division ratio being 80% for the training set, 10% for the validation set, and 10% for the test set;

[0070] Choose a deep learning framework, such as TensorFlow or PyTorch, and install the corresponding library and the selected deep learning framework, making sure the hardware can support the training process;

[0071] Loading the selected large language model;

[0072] Defining tasks performed by the selected large language model, such as SQL parsing, data lineage analysis, etc., and fine-tuning the selected large language model using the training set, wherein the fine-tuning achieves the best effect of the selected large language model by adjusting hyperparameters; the hyperparameters include learning rate and batch size;

[0073] Verifying the selected large language model after fine-tuning on the verification set and testing it on the test set to ensure the generalization ability of the model;

[0074] The selected large language model that has been verified and tested is saved.

[0075] S204: Prepare SQL statements to be analyzed by blood relationship, and use the SQL statements as input of the large language model.

[0076] For example, when the SQL statement to be analyzed by lineage analysis is:

[0077] insert overwrite table ods_mc_meta.dws_it_mc_table_column_d partition(ds='${bizdate}')

[0078] select tt.project_name,tt.type as table_type,tc.project_name||'-'||tc.table_na-me as table_name,count(distinct tc.name)as column_cnt

[0079] from ods_mc_meta.m_column as tc

[0080] inner join ods_mc_meta.m_table as tt

[0081] on tt.name=tc.table_name

[0082] and tt.project_name=tc.project_name

[0083] and tt.type in('TABLE','VIEW')

[0084] and tt.FROM_UNIXTIME(last_modify_time)>DATETIME'2024-01-0100:00:00'

[0085] and tt.ds=max_pt('ods_mc_meta.m_table')

[0086] where tc.ds=max_pt('ods_mc_meta.m_column')

[0087] group by tt.project_name,tt.type,tc.project_name||'-'||tc.table_name;

[0088] Extract data from the m_column and m_table tables, perform inner joins, filtering, and grouping, and finally insert the processed results into the specified partition in the dws_it_mc_table_column_d table.

[0089] S205: Extract metadata of the table involved in the SQL statement using the large language model.

[0090] Get the metadata of the table involved in the SQL statement, including table name, table comment, field name, field type, field comment, etc. For example, in the previous step, the tables involved in the SQL statement are m_column, m_table, and dws_it_mc_table_column_d. Get their metadata. Take the metadata of the dws_it_mc_table_column_d table as an example:

[0091]

[0092] S206: Construct prompt words.

[0093] Explain the specific context in which the task occurs in the large language model, and provide the large language model with necessary environmental information to ensure that its feedback is closely related and targeted. For example, let the large language model analyze the data lineage of SQL statements to clearly understand the source and transformation process of each field.

[0094] Defining the task goal performed by the large language model helps the model focus on achieving the task goal and improve the relevance and accuracy of the answer. For example, the large language model receives SQL statements and table structure information involved in the SQL statements, analyzes whether the SQL statements and table structure information are complete, and if incomplete, requires re-providing, and then organizes and generates accurate data lineage information;

[0095] Indicating the writing style of the large language model, so as to guide the large language model to respond in an expected form and terminology, for example, the large language model outputs the result in a table, and the output table contains a number, target field information, source field information and processing method;

[0096] Defining the attitude and emotional tone of the large language model response to ensure that the model feedback can match the required emotional atmosphere or context;

[0097] Define specific groups that receive responses from the large language model, and customize the model's responses based on audience characteristics to ensure that the information delivered is both appropriate and easy for the group to understand;

[0098] The structure of the large language model response is pre-set to ensure that the model output can seamlessly meet the needs of subsequent processing flows, such as enumeration, JSON format data, and professional analysis reports. Especially for those application scenarios that require processing model feedback programmatically, the JSON format is usually considered the optimal choice.

[0099] S207: The large language model outputs a lineage analysis result including the SQL statement according to the constructed prompt words based on the SQL statement and metadata of the table involved in the SQL statement.

[0100] The prompt words constructed in the previous step are input into the large language model, and the large language model outputs the blood relationship analysis result. As shown in Table 1, Table 1 shows that the output result of the large language model is output in a table form, and the output table contains the number, target field information, source field information and processing method.

[0101] Table 1 Example of model output results

[0102]

[0103]

[0104] Among them, NUMBER starts numbering from 1 and increases by 1 each time; each row in TARGET_COLUMN outputs only one target field, which needs to conform to the field order of the target table. The format is target database.target table.target field, and the value can only be obtained from the table structure of the target table; the format of SOURCE_COLUMN is source database.source table.source field. If a target field is generated by multiple source fields, then multiple source fields must be listed and separated by commas, and the value can only be obtained from the table structure of the source table; PROCESS_TYPE is selected from the following values ​​according to the processing logic: constant assignment, single source field processing, and multiple source field processing.

[0105] Through training, learning and tuning, the large language model can adapt to various complex SQL statements such as multi-table joins, subqueries and filtering, and the SQL dialects and custom extension functions of different database systems. It learns from actual SQL statements instead of relying on predefined grammar rules. It does not require database experts and complex code programming. People focus on relatively low-threshold tasks such as collecting SQL syntax documents for various database systems, training and tuning large language models, and writing more complete prompt words.

[0106] Based on the data lineage acquisition method based on a large language model introduced in the previous embodiment, the present application also provides a data lineage acquisition device based on a large language model. Figure 3 Figure 1 is a schematic diagram of the structure of the device. Figure 3 As shown, the data lineage acquisition device based on the large language model includes:

[0107] A lineage analysis statement acquisition module 301 is used to acquire SQL statements to be analyzed by lineage analysis, and use the SQL statements as input of a large language model;

[0108] The metadata extraction module 302 is used to extract the metadata of the table involved in the SQL statement by using the large language model; wherein the metadata of the table includes a table creation statement; the table creation statement includes a table name, a table remark, a field name, a field type and a field remark;

[0109] The lineage analysis result output module 303 is used for the large language model to output the lineage analysis result containing the SQL statement according to the SQL statement and the metadata of the table involved in the SQL statement according to the constructed prompt words; wherein the constructed prompt words contain the key content involved in the lineage analysis result required to be output, so as to guide the large language model to generate the expected response process; the lineage analysis result of the SQL statement includes the source and transformation process of each field in the SQL statement.

[0110] In an optional implementation, the data lineage acquisition device based on the large language model also includes: a large language model selection module, a sample data preparation module and a large language model fine-tuning module.

[0111] The large language model selection module is used to select a large language model whose actual score value is greater than a preset score value through comprehensive evaluation. The large language model selection module is specifically used to:

[0112] An evaluation data set is constructed using SQL statements that are actually running online; the evaluation data set includes the SQL statements and the source table, source field, target table, target field, and processing method of the SQL statements; the SQL statements cover multiple types of writing scenarios in terms of SQL type distribution, and cover multiple levels of complexity in terms of SQL statement complexity;

[0113] Based on the evaluation data set, the existing large language model is trained, and the performance parameters and related capability indicators of the existing large language model are weighted according to the task requirements of lineage analysis, and the actual score value of the existing large language model is calculated, and the large language model with an actual score value greater than the preset score value is selected as the large language model for data lineage acquisition; the performance parameters of the existing large language model include accuracy and recall rate; the related capability indicators include complexity handling capability, robustness and error analysis, time series analysis and interpretability.

[0114] The sample data preparation module is specifically used for:

[0115] Collect and organize SQL statement samples corresponding to the database system; the SQL statement samples include SQL syntax descriptions and examples, function usage instructions and examples, and SQL statements that have been executed in the database; the SQL syntax descriptions and examples shall be based on those published on the official website of each database system; the SQL statements that have been executed in the database shall be based on those that have been successfully executed.

[0116] The large language model fine-tuning module is used to fine-tune the selected large language model using the sample data, specifically for:

[0117] Removing duplicate and invalid samples from the sample data, marking the relationships between tables and fields in the SQL statements contained in the sample data, and dividing the data set consisting of the sample data into a training set, a validation set, and a test set;

[0118] Select a deep learning framework, and install the library corresponding to the selected deep learning framework and the selected deep learning framework;

[0119] Loading the selected large language model;

[0120] Defining tasks to be performed by the selected large language model, and fine-tuning the selected large language model using the training set; the tasks include SQL parsing and data lineage analysis; the fine-tuning achieves the best effect of the selected large language model by adjusting hyperparameters; the hyperparameters include learning rate and batch size;

[0121] Verifying the selected large language model after fine-tuning on the validation set and testing it on the test set;

[0122] The selected large language model that has been verified and tested is saved.

[0123] In an optional implementation, the data lineage acquisition device based on the large language model includes a prompt word construction module, which is specifically used to:

[0124] Describe the specific context in which the task occurs in the large language model;

[0125] Defining a task objective to be performed by the large language model;

[0126] indicating a writing style of the large language model;

[0127] Defining the attitude and emotional color of the large language model response;

[0128] Defining a specific group that receives responses from the large language model;

[0129] The structural form of the large language model response is preset.

[0130] In addition, an embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the program is executed by a processor, a data lineage acquisition method based on a large language model as described in any of the method embodiments is implemented.

[0131] In addition, an embodiment of the present application also provides a processor, which is used to run a computer program. When the program is running, it executes the data lineage acquisition method based on a large language model as introduced in any implementation method of the aforementioned method embodiment.

[0132] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0133] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.< / length> < / str>

Claims

1. A method for acquiring data lineage based on a large language model, characterized in that: include: Obtaining a SQL statement to be analyzed by lineage analysis, and using the SQL statement as input of a large language model; The large language model is used to extract metadata of the table involved in the SQL statement; wherein the metadata of the table includes a table creation statement; the table creation statement includes a table name, a table remark, a field name, a field type, and a field remark; The large language model outputs a lineage analysis result including the SQL statement according to the constructed prompt words based on the SQL statement and the metadata of the table involved in the SQL statement; wherein the constructed prompt words include key content involved in the lineage analysis result required to be output, so as to guide the large language model to generate an expected response process; the lineage analysis result of the SQL statement includes the source and transformation process of each field in the SQL statement.

2. The method according to claim 1, characterized in that Before obtaining the SQL statement to be analyzed by blood relationship, it also includes: A large language model with an actual score value greater than a preset score value is selected through comprehensive evaluation, sample data is prepared, and the selected large language model is fine-tuned using the sample data.

3. The method according to claim 2, characterized in that The step of selecting a large language model whose actual score value is greater than a preset score value through comprehensive evaluation includes: An evaluation data set is constructed using SQL statements that are actually running online; the evaluation data set includes the SQL statements and the source table, source field, target table, target field, and processing method of the SQL statements; the SQL statements cover multiple types of writing scenarios in terms of SQL type distribution, and cover multiple levels of complexity in terms of SQL statement complexity; Based on the evaluation data set, the existing large language model is trained, and the performance parameters and related capability indicators of the existing large language model are weighted according to the task requirements of lineage analysis, and the actual score value of the existing large language model is calculated, and the large language model with an actual score value greater than the preset score value is selected as the large language model for data lineage acquisition; the performance parameters of the existing large language model include accuracy and recall rate; the related capability indicators include complexity handling capability, robustness and error analysis, time series analysis and interpretability.

4. The method according to claim 2, characterized in that: The preparing sample data comprises: Collect and organize SQL statement samples corresponding to the database system; the SQL statement samples include SQL syntax descriptions and examples, function usage instructions and examples, and SQL statements that have been executed in the database; the SQL syntax descriptions and examples shall be based on those published on the official website of each database system; the SQL statements that have been executed in the database shall be based on those that have been successfully executed.

5. The method according to claim 2, characterized in that: The fine-tuning of the selected large language model using the sample data includes: Removing duplicate and invalid samples from the sample data, marking the relationships between tables and fields in the SQL statements contained in the sample data, and dividing the data set consisting of the sample data into a training set, a validation set, and a test set; Select a deep learning framework, and install the library corresponding to the selected deep learning framework and the selected deep learning framework; Loading the selected large language model; Defining tasks to be performed by the selected large language model, and fine-tuning the selected large language model using the training set; the tasks include SQL parsing and data lineage analysis; the fine-tuning achieves the best effect of the selected large language model by adjusting hyperparameters; the hyperparameters include learning rate and batch size; Verifying the selected large language model after fine-tuning on the validation set and testing it on the test set; The selected large language model that has been verified and tested is saved.

6. The method according to claim 1, characterized in that The construction method of the prompt word includes: Describe the specific context in which the task occurs in the large language model; Defining a task objective to be performed by the large language model; indicating a writing style of the large language model; Defining the attitude and emotional color of the large language model response; Defining a specific group that receives responses from the large language model; The structural form of the large language model response is preset.

7. A data lineage acquisition device based on a large language model, characterized in that: include: A lineage analysis statement acquisition module is used to acquire SQL statements to be analyzed by lineage, and use the SQL statements as input of the large language model; A metadata extraction module, used to extract metadata of the table involved in the SQL statement using the large language model; wherein the metadata of the table includes a table creation statement; the table creation statement includes a table name, a table remark, a field name, a field type, and a field remark; A lineage analysis result output module is used for the large language model to output the lineage analysis result containing the SQL statement according to the constructed prompt words based on the SQL statement and the metadata of the table involved in the SQL statement; wherein the constructed prompt words include key content involved in the lineage analysis result required to be output, so as to guide the large language model to generate an expected response process; the lineage analysis result of the SQL statement includes the source and transformation process of each field in the SQL statement.

8. A data lineage acquisition device based on a large language model, characterized in that: Also includes: Large language model selection module, sample data preparation module and large language model fine-tuning module.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the program is executed by the processor, the method for acquiring data lineage based on a large language model as described in any one of claims 1 to 6 is implemented.

10. A processor, characterized in that: Used to run a computer program, which, when running, executes the data lineage acquisition method based on a large language model as described in any one of claims 1 to 6.

Images