Method, device and equipment for extracting field consanguinity of SQL (Structured Query Language) statement and storage medium

Abstract

The invention discloses a field consanguinity extraction method and device for SQL statements, equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps that corresponding field consanguinity is labeled for a plurality of first SQL statements to serve as training data; cue words are designed according to the SQL statements; training a specified weight matrix in the large language model by using the training data and the cue words to obtain a trained large language model; and extracting the field blood relationship of the second SQL statement by utilizing the trained large language model. According to the method, the training data and the cue words are used for training to obtain the large language model, so that the large language model learns and understands the field blood relationship in the SQL statement, the problem that the related technology is difficult to adapt to the diversity and complexity of the SQL statement is solved, and the accuracy and efficiency of field blood relationship extraction are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and storage medium for extracting field lineage of an SQL statement. Background Art

[0002] Field lineage refers to the dependency between data sources and data targets. Field lineage reflects the source and flow of data, and is of great significance to data governance and data quality management. SQL statements are a commonly used data processing language, and SQL statements contain rich field lineage information, but related technologies are difficult to adapt to the diversity and complexity of SQL statements, resulting in the inability to efficiently extract field lineage from SQL statements. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a method, device, equipment and storage medium for extracting field lineage of SQL statements, so as to efficiently extract field lineage of SQL statements.

[0004] To achieve the above purpose, an embodiment of the present application provides a method for extracting field lineage from an SQL statement, the method comprising the following steps:

[0005] Marking the corresponding field relationships of the multiple first SQL statements as training data;

[0006] Design prompt words according to SQL statements;

[0007] Using the training data and the prompt words to train a weight matrix specified in a large language model to obtain a trained large language model;

[0008] The trained large language model is used to extract the field lineage of the second SQL statement.

[0009] In some embodiments, the step of respectively labeling corresponding field lineages of the plurality of first SQL statements as training data comprises the following steps:

[0010] Obtain each of the first SQL statements;

[0011] According to the syntax rules of SQL statements, the corresponding field lineages are respectively marked for each of the first SQL statements;

[0012] Each of the first SQL statements annotated with field lineage is converted into the training data of the large language model.

[0013] In some embodiments, the obtaining of each of the first SQL statements comprises the following steps:

[0014] A plurality of different types of the first SQL statements are obtained; wherein the first SQL statements of different types have different attributes, and the attributes include whether there is a group by, whether there is a where, whether there is field multiplication or addition, and the number of table processing.

[0015] In some embodiments, the step of marking the corresponding field lineage of each of the first SQL statements according to the grammatical rules of the SQL statements comprises the following steps:

[0016] Extracting a plurality of fields from each of the first SQL statements according to the syntax rules of the SQL statements; wherein the fields include at least one of a string, a table, a function or a condition;

[0017] The corresponding field lineage is respectively marked on each of the fields according to the metadata of each of the first SQL statements and the business rules corresponding to each of the first SQL statements.

[0018] In some embodiments, the step of using the training data and the prompt words to train a specified weight matrix in a large language model to obtain a trained large language model comprises the following steps:

[0019] Adding an additional parameter matrix in parallel to the specified weight matrix in the large language model; wherein the rank of the parameter matrix is ​​smaller than the rank of the weight matrix;

[0020] The parameter matrix in the large language model is trained using the LoRA method according to the training data and the prompt word to obtain the trained large language model.

[0021] In some embodiments, the method of using the LoRA method and training the parameter matrix in the large language model according to the training data and the prompt word to obtain the trained large language model includes the following steps:

[0022] Determine the LoRA parameters of the LoRA method; wherein the LoRA parameters include a learning rate, a rank of the parameter matrix, a scaling factor of the parameter matrix, a dropout probability, a maximum length of an input sequence, a maximum length of an output sequence, a batch size of a development set, a number of steps for gradient accumulation, a maximum number of training steps, an interval for saving the large language model parameters, and a maximum sequence length of the large language model;

[0023] The parameter matrix in the large language model is trained according to the LoRA parameters, the training data and the prompt words to obtain the trained large language model.

[0024] In some embodiments, the method further comprises the following steps:

[0025] The metadata operated by the second SQL statement is post-processed using the field lineage of the second SQL statement.

[0026] To achieve the above purpose, another aspect of the embodiment of the present application provides a field lineage extraction device for SQL statements, the device comprising:

[0027] A training data acquisition unit, used to respectively mark corresponding field relationships of a plurality of first SQL statements as training data;

[0028] A prompt word design unit, used to design prompt words according to SQL statements;

[0029] A model training unit, used to train a weight matrix specified in a large language model using the training data and the prompt words to obtain a trained large language model;

[0030] The field lineage extraction unit is used to extract the field lineage of the second SQL statement using the trained large language model.

[0031] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0032] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0033] The embodiments of the present application include at least the following beneficial effects:

[0034] The present application can respectively mark the corresponding field lineages of multiple first SQL statements as training data; design prompt words according to the SQL statements; use the training data and prompt words to train the weight matrix specified in the large language model to obtain the trained large language model; and use the trained large language model to extract the field lineage of the second SQL statement. The large language model is trained using the training data and prompt words, so that the large language model learns and understands the field lineage in the SQL statement, thereby overcoming the problem that the related technology is difficult to adapt to the diversity and complexity of SQL statements, and improving the accuracy and efficiency of field lineage extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 A flowchart of a method for extracting field lineage from an SQL statement provided in an embodiment of the present application;

[0037] Figure 2 Schematic diagram of the LoRA method provided in the embodiment of the present application;

[0038] Figure 3 A schematic diagram of the structure of a field lineage extraction device for an SQL statement provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0041] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0042] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0044] Before describing the embodiments of the present application in detail, the related technologies involved in the embodiments of the present application are first described as follows:

[0045] Field lineage refers to the dependency between the data source and the data target. Field lineage reflects the source and flow of data, which is of great significance for data governance and data quality management. SQL statements are a commonly used data processing language, and SQL statements contain rich field lineage information, such as the correspondence between the result column in the SELECT statement and the table and field in the FROM clause, as well as the source of fields in nested subqueries and function expressions. Therefore, how to extract field lineage from SQL statements is an important research issue.

[0046] At present, there are some related technologies that can parse the field lineage in SQL statements, such as SQLFlow2 or SQLParser3. Related technologies usually extract field lineage based on SQL parser, that is, according to the grammatical rules of SQL language, SQL statements are parsed into abstract syntax tree (AST), and then field lineage is extracted from AST. Related technologies based on grammatical analysis are difficult to adapt to the diversity and complexity of SQL statements and the actual needs of data governance.

[0047] Therefore, the embodiments of the present application provide a method, device, equipment and storage medium for extracting field lineage of SQL statements. The technical solution of the present application includes: respectively marking the corresponding field lineages of multiple first SQL statements as training data; designing prompt words according to the SQL statements; using the training data and prompt words to train the weight matrix specified in the large language model to obtain the trained large language model; using the trained large language model to extract the field lineage of the second SQL statement. The large language model is trained using the training data and prompt words, so that the large language model learns and understands the field lineage in the SQL statement, thereby overcoming the problem that the related technology is difficult to adapt to the diversity and complexity of SQL statements, and improving the accuracy and efficiency of field lineage extraction.

[0048] The embodiment of the present application provides a method for extracting field lineage from an SQL statement, and relates to the field of data processing technology. The method for extracting field lineage provided in the embodiment of the present application can be applied to a terminal, or to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the field lineage extraction method, etc., but is not limited to the above forms.

[0049] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0050] Reference Figure 1 The present application embodiment provides a method for extracting field lineage from a SQL statement. The method may include but is not limited to steps S100 to S130, as follows:

[0051] S100: Labeling corresponding field lineages of a plurality of first SQL statements respectively as training data.

[0052] It is understandable that each first SQL statement may correspond to a different field lineage, so this embodiment may mark the corresponding field lineage for each first SQL statement.

[0053] Specifically, the training data of this embodiment includes training samples and training labels, wherein each first SQL statement serves as a training sample, and the field lineage annotated by each first SQL statement serves as a training label.

[0054] Furthermore, S100 may include S101 to S103:

[0055] S101: Obtain each of the first SQL statements.

[0056] It can be understood that, in this embodiment, multiple SQL statements can be obtained as respective first SQL statements.

[0057] Furthermore, S101 may include:

[0058] A plurality of different types of the first SQL statements are obtained; wherein the first SQL statements of different types have different attributes, and the attributes include whether there is a group by, whether there is a where, whether there is field multiplication or addition, and the number of table processing.

[0059] Exemplarily, to improve the training effect, the number of first SQL statements of each type obtained may be plural, and the numbers obtained may be the same or different. For example, 10 first SQL statements of each type may be obtained.

[0060] S102: Mark the corresponding field lineage of each of the first SQL statements according to the grammatical rules of the SQL statement.

[0061] Furthermore, S102 may include S1021-S1022:

[0062] S1021: extracting a plurality of fields from each of the first SQL statements according to the syntax rules of the SQL statements; wherein the fields include at least one of a string, a table, a function or a condition;

[0063] S1022: Mark the corresponding field lineage for each of the fields according to the metadata of each of the first SQL statements and the business rules corresponding to each of the first SQL statements.

[0064] It is understandable that this embodiment can determine the character strings, tables, functions or conditions included in each first SQL statement according to the grammatical rules of the SQL statement, and mark the corresponding field lineage for each field according to the corresponding metadata and business rules.

[0065] S103: Convert each of the first SQL statements annotated with field lineage into the training data of the large language model.

[0066] It is understandable that, in this embodiment, each first SQL statement annotated with field lineage can be converted into training data of the large language model according to the training data requirement of the large language model.

[0067] S110: Design prompt words according to the SQL statement.

[0068] Specifically, a prompt is a medium for people to interact with a large language model, usually consisting of two parts: instructions and content. The prompt can be used to inform the large language model of the purpose that the user wants to achieve, and provide some relevant information or examples, that is, the prompt is designed to enable the large language model to better understand and complete the given task. The prompt examples designed in this embodiment are as follows:

[0069] "Please extract the field lineage in the SQL statement, and require that the target table fields and source table fields in the input SQL statement be mapped, including functions. The input SQL statement is as follows:".

[0070] S120: Using the training data and the prompt words to train a weight matrix specified in a large language model to obtain a trained large language model.

[0071] It is understandable that in this embodiment, some parameters of the large language model can be frozen, a specified weight matrix can be retained, and then the specified weight matrix can be trained using training data and prompt words.

[0072] Further, S120 may include:

[0073] S121: Adding an additional parameter matrix in parallel to the specified weight matrix in the large language model; wherein the rank of the parameter matrix is ​​smaller than the rank of the weight matrix.

[0074] Specifically, when the rank value is much smaller than the original parameter dimension, the number of parameters in the newly added parameter matrix (low-rank matrix) is also small. Therefore, this embodiment can train very few parameters, that is, can obtain a large language model with better extraction effect.

[0075] S122: Using the LoRA method and according to the training data and the prompt word, the parameter matrix in the large language model is trained to obtain the trained large language model.

[0076] Specifically, LoRA (Low-Rank Adaptation) is a model fine-tuning method that allows the original model parameters to be kept unchanged while adapting to new tasks or data by introducing low-rank matrix updates. The LoRA method is particularly suitable for large language models, such as GPT-3, by reducing the number of parameters that need to be updated, significantly reducing computing resources and storage requirements while maintaining the performance of large language models.

[0077] Furthermore, S122 may include S1221-S1222:

[0078] S1221: Determine the LoRA parameters of the LoRA method; wherein the LoRA parameters include a learning rate, a rank of the parameter matrix, a scaling factor of the parameter matrix, a dropout probability, a maximum length of an input sequence, a maximum length of an output sequence, a batch size of a development set, a number of steps for gradient accumulation, a maximum number of training steps, an interval for saving parameters of the large language model, and a maximum sequence length of the large language model.

[0079] S1222: Train the parameter matrix in the large language model according to the LoRA parameters, the training data, and the prompt words to obtain the trained large language model.

[0080] S130: Extracting field lineage of the second SQL statement using the trained large language model.

[0081] It is understandable that in this embodiment, any SQL statement can be used as the second SQL statement, and then the second SQL statement can be input into the trained large language model, and the trained large language model can be used to extract and output the field lineage of the second SQL statement.

[0082] Furthermore, the embodiment of the present application may also include the following steps:

[0083] The metadata operated by the second SQL statement is post-processed using the field lineage of the second SQL statement.

[0084] It is understandable that the embodiment of the present application can also perform post-processing such as data governance and data quality management on the metadata of the second SQL statement operation according to the field lineage of the second SQL statement.

[0085] Next, the solution of the embodiment of the present application will be introduced and explained in conjunction with specific application examples:

[0086] Specifically, this embodiment may include the following steps:

[0087] Step 1: Collect SQL statements, perform syntax analysis on each SQL statement, and perform field lineage annotation, and then generate training data. For example, the definition of field lineage is shown in Table 1:

[0088] Table 1

[0089]

[0090]

[0091] At the same time, in order to ensure the training effect of the large language model, the selected SQL statements can include different types. Exemplarily, the SQL statements used to train the large language model in this embodiment are shown in Table 2:

[0092] Table 2

[0093]

[0094] Specifically, this embodiment can collect SQL statements for field lineage annotation, perform grammatical analysis on each SQL statement, extract fields such as strings, tables, functions, conditions, etc., and annotate each field with lineage according to metadata and business rules, that is, indicate the data source and target of the field, as well as the dependency relationship between each data, and then convert the SQL statements annotated with field lineage into training data according to the training data requirements of the large language model.

[0095] Step 2: Use the LoRA method to fine-tune the large language model. Exemplarily, the large language model of this embodiment can use ChatGLM3.0-7B-Base. LoRA is the abbreviation of Low-Rank Adaptation. The basic principle of the LoRA method is to add additional low-rank matrices to the specified parameters (weight matrix) in parallel on the large language model, and during the training process of the large language model, only train the parameters of the additional parallel low-rank matrices and freeze other parameters. When the rank value is much smaller than the original parameter dimension, the amount of newly added low-rank matrix parameters is also very small. When fine-tuning the model for downstream tasks, very few parameters can be trained, but better performance results can be obtained. The schematic diagram of the LoRA method of this embodiment is shown in the figure. Figure 2 As shown, the LoRA parameters used in this embodiment are shown in Table 3:

[0096] Table 3

[0097]

[0098]

[0099]

[0100] Specifically, this embodiment can perform LoRA fine-tuning training adapted to the SQL field lineage extraction task based on the pre-weighted ChatGLM3.0-7B-Base. The LoRA method can add some learnable parameters while keeping the parameters of ChatGLM3.0-7B-Base unchanged, thereby improving the effect of ChatGLM3.0-7B-Base in extracting field lineage.

[0101] Step 3: Design prompts to guide ChatGLM3.0-7B-Base to complete the task. Prompts are a medium for people to interact with large language models, usually consisting of instructions and content. Prompts can be used to inform the large language model of the purpose the user wants to achieve, and provide some relevant information or examples, that is, design prompts to enable the large language model to better understand and complete the given task. The prompts designed in this embodiment are as follows:

[0102] "Please extract the field lineage in the SQL statement, and require that the target table fields and source table fields in the input SQL statement be mapped, including functions. The input SQL statement is as follows:".

[0103] Beneficial effects of this embodiment:

[0104] This embodiment provides a method for field lineage extraction of SQL statements based on a large language model. The trained large language model can effectively understand and meet the diverse and complex field lineage extraction of SQL statements. The parameter scale of ChatGLM3.0-7B-Base used in this embodiment is 6B, which has low hardware requirements for the image processor and has strong applicability.

[0105] Reference Figure 3 The embodiment of the present application further provides a field lineage extraction device for SQL statements, which can implement the above-mentioned field lineage extraction method for SQL statements, and the device includes:

[0106] A training data acquisition unit, used to respectively mark corresponding field relationships of a plurality of first SQL statements as training data;

[0107] A prompt word design unit, used to design prompt words according to SQL statements;

[0108] A model training unit, used to train a weight matrix specified in a large language model using the training data and the prompt words to obtain a trained large language model;

[0109] The field lineage extraction unit is used to extract the field lineage of the second SQL statement using the trained large language model.

[0110] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0111] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above field lineage extraction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0112] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0113] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0114] The processor 401 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0115] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 402, and the processor 401 calls and executes the field lineage extraction method of the SQL statement of the embodiment of this application;

[0116] Input / output interface 403, used to implement information input and output;

[0117] Communication interface 404, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0118] Bus 405 , which transmits information between various components of the device (e.g., processor 401 , memory 402 , input / output interface 403 , and communication interface 404 );

[0119] The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via the bus 405 .

[0120] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the field lineage extraction method of the above-mentioned SQL statement.

[0121] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0122] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0123] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0124] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0125] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0127] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0128] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0129] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0130] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0133] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for extracting field lineage from SQL statements, characterized in that: The method comprises the following steps: Marking the corresponding field relationships of the multiple first SQL statements as training data; Design prompt words according to SQL statements; Using the training data and the prompt words to train a weight matrix specified in a large language model to obtain a trained large language model; The trained large language model is used to extract the field lineage of the second SQL statement.

2. The method for extracting field lineage from an SQL statement according to claim 1, characterized in that: The step of respectively marking corresponding field lineages of the plurality of first SQL statements as training data comprises the following steps: Obtain each of the first SQL statements; According to the syntax rules of SQL statements, the corresponding field lineages are respectively marked for each of the first SQL statements; Each of the first SQL statements annotated with field lineage is converted into the training data of the large language model.

3. The method for extracting field lineage from an SQL statement according to claim 2, characterized in that: The obtaining of each of the first SQL statements comprises the following steps: A plurality of different types of the first SQL statements are obtained; wherein the first SQL statements of different types have different attributes, and the attributes include whether there is a group by, whether there is a where, whether there is field multiplication or addition, and the number of table processing.

4. The method for extracting field lineage from an SQL statement according to claim 2, characterized in that: The step of marking the corresponding field lineage of each of the first SQL statements according to the grammatical rules of the SQL statements comprises the following steps: Extracting a plurality of fields from each of the first SQL statements according to the syntax rules of the SQL statements; wherein the fields include at least one of a string, a table, a function or a condition; The corresponding field lineage is respectively marked on each of the fields according to the metadata of each of the first SQL statements and the business rules corresponding to each of the first SQL statements.

5. The method for extracting field lineage from an SQL statement according to claim 1, characterized in that: The step of using the training data and the prompt words to train a weight matrix specified in the large language model to obtain a trained large language model comprises the following steps: Adding an additional parameter matrix in parallel to the specified weight matrix in the large language model; wherein the rank of the parameter matrix is ​​smaller than the rank of the weight matrix; The parameter matrix in the large language model is trained using the LoRA method according to the training data and the prompt word to obtain the trained large language model.

6. The method for extracting field lineage from an SQL statement according to claim 5, characterized in that: The method of using the LoRA method and training the parameter matrix in the large language model according to the training data and the prompt word to obtain the trained large language model includes the following steps: Determine the LoRA parameters of the LoRA method; wherein the LoRA parameters include a learning rate, a rank of the parameter matrix, a scaling factor of the parameter matrix, a dropout probability, a maximum length of an input sequence, a maximum length of an output sequence, a batch size of a development set, a number of steps for gradient accumulation, a maximum number of training steps, an interval for saving the large language model parameters, and a maximum sequence length of the large language model; The parameter matrix in the large language model is trained according to the LoRA parameters, the training data and the prompt words to obtain the trained large language model.

7. A method for extracting field lineage from an SQL statement according to any one of claims 1 to 6, characterized in that: The method further comprises the following steps: The metadata operated by the second SQL statement is post-processed using the field lineage of the second SQL statement.

8. A field lineage extraction device for SQL statements, characterized in that: The device comprises: A training data acquisition unit, used to respectively mark corresponding field relationships of a plurality of first SQL statements as training data; A prompt word design unit, used to design prompt words according to SQL statements; A model training unit, used to train a weight matrix specified in a large language model using the training data and the prompt words to obtain a trained large language model; The field lineage extraction unit is used to extract the field lineage of the second SQL statement using the trained large language model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the field lineage extraction method of an SQL statement as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a method for extracting field lineage from an SQL statement according to any one of claims 1 to 7 is implemented.

Images