Data blood relationship analysis method and device, electronic equipment and storage medium
By constructing a prompt word and lineage parsing model based on SQL syntax structure features, the problem of insufficient SQL file parsing depth is solved, and lineage parsing results with higher accuracy and completeness are achieved.
Patent Information
- Application Number
- CN202610168769.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-05
- Publication Date
- 2026-05-15
AI Technical Summary
In existing technologies, the lineage parsing method for SQL files suffers from insufficient parsing depth due to syntactic complexity, failing to reflect the processing details of the data processing logic, resulting in low parsing accuracy and poor completeness of the parsing results.
Construct prompt words based on the syntactic structure features of a structured query language, including multiple operators, their associated parameters, and output formats. Input these prompt words into a lineage resolution model to obtain lineage resolution results. Improve the resolution depth and accuracy through iterative optimization and verification instructions.
By guiding the pedigree analysis model to focus on the details of data processing logic, the insufficient parsing accuracy caused by syntactic complexity is avoided, thereby improving the data integrity and accuracy of the pedigree analysis results.
Smart Images
Figure CN122045224A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of financial technology and data management, and in particular to a data lineage analysis method, apparatus, electronic device, and storage medium. Background Technology
[0002] During the data development and governance process, a large number of technical documents in Structured Query Language (SQL) are generated. Managing the lineage of these technical documents is a key step in understanding and managing these data assets.
[0003] In existing technologies, the parsing of SQL files is usually based on static syntax tree analysis. This method first extracts the SQL statements from different SQL files into a syntax tree according to preset rules. Then, based on the parameter information in the syntax tree, it further generates statement-level table field mapping relationships. Finally, by reconstructing the SQL mapping relationships, it extracts table mappings and field mappings, thereby forming the data lineage of the SQL files.
[0004] However, due to the complexity of SQL file syntax, this lineage resolution method often results in insufficient parsing depth, failing to reflect the processing details of the data processing logic, thus leading to low parsing accuracy and poor completeness of the lineage resolution results. Summary of the Invention
[0005] This invention provides a data lineage analysis method, apparatus, electronic device, and storage medium to solve the problem of low parsing accuracy of SQL file lineage analysis results.
[0006] According to another aspect of the present invention, a data lineage analysis method is provided, comprising: Obtain the prompt words corresponding to the set of files to be parsed; wherein, the set of files includes multiple Structured Query Language (SCL) files; the prompt words are constructed based on the syntactic structure features of SCL; the prompt words include multiple operators and the associated parameters and output format of each operator; The prompt words are input into the lineage resolution model so that the lineage resolution model can obtain the lineage resolution results of each of the structured query language files based on the prompt words.
[0007] The step of inputting the prompt word into the lineage resolution model so that the lineage resolution model can obtain the lineage resolution results of each of the structured query language files based on the prompt word includes: obtaining the abstract syntax tree of the current structured query language file, and obtaining the initial resolution result of the current structured query language file based on the abstract syntax tree; embedding the initial resolution result as context information into the prompt word; and inputting the prompt word into the lineage resolution model so that the lineage resolution model can obtain the lineage resolution results of each of the structured query language files based on the prompt word.
[0008] After obtaining the initial parsing result of the current structured query language file based on the abstract syntax tree, the method further includes: comparing the lineage parsing result of the current structured query language file with the initial parsing result; and when it is determined that there is a missing table name in the lineage parsing result, updating the missing table name to the prompt word; and returning to execute the operation of inputting the prompt word into the lineage parsing model for processing, so as to output the updated lineage parsing result.
[0009] The data lineage analysis method further includes: obtaining the lineage analysis results of a test sample set based on the prompt word through the lineage analysis model; obtaining the lineage analysis deviation based on the lineage analysis results of the test sample set and the expected lineage analysis results; obtaining a matching update strategy based on the lineage analysis deviation, updating the prompt word according to the update strategy, and performing capacity expansion processing on the test sample set; returning to execute the operations of obtaining the lineage analysis results of the test sample set based on the prompt word through the lineage analysis model, and obtaining the lineage analysis deviation based on the lineage analysis results of the test sample set and the expected lineage analysis results, until the lineage analysis results of the test sample set no longer have lineage analysis deviation.
[0010] The step of obtaining the bloodline analysis deviation based on the bloodline analysis results and expected bloodline analysis results of the test sample set specifically includes: obtaining the bloodline analysis deviation through a rule engine and a discrimination model based on the bloodline analysis results and expected bloodline analysis results of the test sample set.
[0011] The step of inputting the prompt words into the lineage resolution model so that the lineage resolution model can obtain the lineage resolution results of each of the structured query language files based on the prompt words specifically includes: adding the validation instructions based on the abstract syntax tree into the prompt words, and inputting the prompt words into the lineage resolution model so as to obtain multiple lineage resolution results of each of the structured query language files through the lineage resolution model; among the multiple lineage resolution results, the target lineage resolution result with the highest confidence score is taken as the actual lineage resolution result.
[0012] According to another aspect of the present invention, a data lineage analysis apparatus is provided, comprising: The prompt word acquisition module is used to acquire prompt words corresponding to the set of files to be parsed; wherein, the set of files includes multiple structured query language files; the prompt words are constructed based on the syntactic structure features of structured query language; the prompt words include multiple operators and the associated parameters and output format of each operator; The parsing and execution module is used to input the prompt words into the lineage parsing model, so that the lineage parsing model can obtain the lineage parsing results of each of the structured query language files based on the prompt words.
[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data lineage analysis method according to any embodiment of the present invention.
[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data lineage resolution method according to any embodiment of the present invention.
[0015] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the data lineage analysis method described in any embodiment of the present invention.
[0016] The technical solution of this invention constructs prompt words based on the syntactic structure features of Structured Query Language (SCL). These prompt words include multiple operators, their associated parameters, and output formats. After obtaining the prompt words corresponding to the set of files to be parsed, the prompt words are input into a lineage parsing model. This allows the lineage parsing model to obtain the lineage parsing results for each SCL file based on the prompt words. By incorporating multiple SCL operators, their associated parameters, and output formats into the prompt words, the prompt words reflect the processing details of the data processing logic. This guides the lineage parsing model to perform deep parsing operations, avoiding insufficient parsing accuracy due to syntactic complexity and improving the data integrity of the lineage parsing results.
[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart of a data lineage analysis method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of another data lineage analysis method provided in Embodiment 2 of the present invention; Figure 3 This is a flowchart of another data lineage analysis method provided in Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of a data lineage analysis device according to Embodiment 4 of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device that implements the data lineage analysis method of this invention. Detailed Implementation
[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0022] Example 1 Figure 1This is a flowchart of a data lineage parsing method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where prompt words are input into a lineage parsing model to obtain the lineage parsing results of each structured query language file in a file set. This method can be executed by a data lineage parsing device, which can be implemented in hardware and / or software. The data lineage parsing device can be configured in an electronic device. Figure 1 As shown, the method includes: S101. Obtain the prompt words corresponding to the set of files to be parsed; wherein, the set of files includes multiple structured query language files; the prompt words are constructed based on the syntactic structure features of the structured query language; the prompt words include multiple operators and the associated parameters and output format of each operator.
[0023] An operator is an abstract description of each operation step in the execution of an SQL statement, and it is also the smallest functional unit of that execution step. It defines how data is read, filtered, joined, aggregated, etc. Operators form a workflow through input-output relationships. The syntax structure of SQL defines operators used to indicate the direction of data flow, such as SELECT (target field), FROM (data source), JOIN (table join), and WHERE (filter condition). Among them, the target field indicates the specified output field; the data source defines the data source table or subquery; table join is used to connect multiple tables; and the filter condition is used to perform data filtering operations.
[0024] High-frequency operators (i.e., operators whose frequency is greater than or equal to a preset frequency threshold) appearing in historical SQL files can be added as target operators to the prompt words; the associated parameters of the operator include input values and output values; the input value represents the data source processed by the operator, which can be a table or view, the output of other operators, or a specified constant or expression; the output value represents the result data generated after the operator is processed, which can be used as the input of other operators or the final query result.
[0025] In addition, different types of association parameters can be configured for different types of operators. For example, a join condition can be configured for the JOIN operator mentioned above. The join condition represents the association rule between two tables, which determines which rows need to be merged. A result alias can be configured for the SELECT, FROM, or JOIN operators mentioned above. This means configuring temporary names in the columns or tables of the query results of the above operators. This is used to simplify references, avoid ambiguity, and improve readability.
[0026] The output format defines how some or all of the associated parameters of an operator are output. Taking the above technical solution as an example, the output format of the SELECT operator can be defined as "operator name + input value + output value", the output format of the FROM operator can be defined as "operator name + input value + output value + result alias", and the output format of the JOIN operator can be defined as "operator name + input value + output value + join condition + result alias".
[0027] In addition, SQL's syntax structure features define field dependencies and table dependencies. Field dependencies include simple fields (e.g., table.column), computed fields (e.g., expr(col1, col2)), and alias fields (e.g., alias AS new_name). Table dependencies include base tables (e.g., FROM table), subqueries (e.g., FROM (SELECT ...) AS sub), and views (e.g., FROM view). Based on this, keywords related to the above field and table dependencies can also be added to the prompt words to guide the lineage resolution model to focus on the multiple sources of each table and field.
[0028] Optionally, in this embodiment of the invention, after obtaining the prompt words corresponding to the set of files to be parsed, the method further includes: obtaining the dialect features of the structured query language according to the data source of each structured query language file, and embedding the dialect features as context information into the prompt words.
[0029] Specifically, to facilitate the management and operation of SQL files, different types of databases usually extend and customize the standard SQL based on their own system characteristics. These differences form the dialects of SQL. Therefore, the dialect characteristics of SQL files actually refer to the specific implementations and extensions of SQL by different databases (or different types of databases). SQL files in different databases may differ in terms of syntax, functions, and data types.
[0030] The data sources of each SQL file in the set of files to be parsed are aggregated, and the dialect features of each database (or each type of database) are configured in the prompt words. By injecting dialect features, the lineage parsing model's ability to understand non-standard syntax is improved. In particular, different customized prompt words can be generated based on the dialect features of different databases.
[0031] For example, based on the dialect features of different databases, operators such as adding, deleting, and modifying prompt words are performed to generate customized prompt words for different databases. In this way, the customized prompt words of each database and the dialect features of that database are configured together to the lineage analysis model. This allows the lineage analysis model to perform targeted reasoning based on the data source of each SQL file, according to the matched customized prompt words and dialect features, thus enhancing the lineage analysis model's ability to understand SQL files.
[0032] The lineage analysis model is an artificial intelligence (AI) model trained on lineage analysis samples, based on the Large Language Model (LLM). By parsing SQL files, it tracks the data flow path, dependencies, and transformation logic in SQL statements, thereby breaking through the performance bottleneck of traditional SQL parsing methods and providing efficient and accurate data lineage relationships for data processing needs such as data governance, troubleshooting, and compliance auditing.
[0033] S102. Input the prompt words into the lineage resolution model so that the lineage resolution model can obtain the lineage resolution results of each of the structured query language files based on the prompt words.
[0034] The lineage analysis results can be presented in the form of a graph; where fields and tables are configured as nodes in the graph structure, and data flow is configured as edges in the graph structure. Based on the above lineage analysis graph, when data anomalies are detected, problematic SQL statements and upstream tables can be automatically located, and repair suggestions can be generated, thereby achieving intelligent fault attribution and significantly reducing fault troubleshooting time. At the same time, based on the above lineage analysis graph, useless tables and useless fields can be identified, and storage resources can be optimized, thereby achieving data governance.
[0035] By using multiple operators and their associated parameters as prompts, the lineage resolution model is guided to focus on the key data processing logic in the SQL file, ignoring irrelevant semantic information in the code file (such as comments, access control statements, and stored procedure logic). This selective attention guidance mechanism effectively suppresses the problem of the lineage resolution model introducing irrelevant semantic information due to overgeneralization.
[0036] Meanwhile, the output format limits the output range of the lineage resolution model, effectively constraining it to process only lineage-related elements. This not only avoids semantic divergence but also ensures that the output results can be directly parsed and called by downstream systems. Furthermore, by adding the keywords involved in the aforementioned field and table dependencies to the prompt words, the lineage analysis depth of the lineage resolution model is further increased, improving the data quality of the lineage resolution results.
[0037] Optionally, in this embodiment of the invention, the step of inputting the prompt word into the lineage resolution model so that the lineage resolution model obtains the lineage resolution results of each of the structured query language files based on the prompt word includes: obtaining the abstract syntax tree of the current structured query language file, and obtaining the initial parsing result of the current structured query language file according to the abstract syntax tree; embedding the initial parsing result as context information into the prompt word; and inputting the prompt word into the lineage resolution model so that the lineage resolution model obtains the lineage resolution results of each of the structured query language files based on the prompt word.
[0038] Specifically, firstly, regular expressions are used to identify tokens (keywords, identifiers such as table names and field names), operators, delimiters, and literals in the SQL file; secondly, based on predefined SQL syntax rules, an Abstract Syntax Tree (AST) is constructed using the obtained token information, thereby transforming the code logic in the SQL file into a tree-like data structure, so that each node corresponds to a token; finally, structured information, such as table names, lists, and field reference relationships, is obtained by traversing the AST.
[0039] By embedding the initial parsing results as contextual information into prompt words, the lineage resolution model can acquire sufficient prior knowledge when facing complex SQL scenarios (such as nested subqueries, recursion, and dynamic aliases), thereby improving its understanding ability. Thus, by using operators and their associated parameters to guide the lineage resolution model to effectively suppress the introduction of irrelevant semantic information due to overgeneralization, the initial parsing results also guide the lineage resolution model to strengthen its ability to recognize structured information, ensuring the integrity of the lineage resolution results.
[0040] Optionally, in this embodiment of the invention, after obtaining the initial parsing result of the current structured query language file according to the abstract syntax tree, the method further includes: comparing the lineage parsing result of the current structured query language file with the initial parsing result, and when it is determined that there is a missing table name in the lineage parsing result, updating the missing table name to the prompt word, and returning to execute the operation of inputting the prompt word into the lineage parsing model for processing, so as to output the updated lineage parsing result.
[0041] Specifically, although the lineage resolution model has a certain generalization ability, it may still encounter anomalies when dealing with boundary cases. Therefore, a dual-channel verification architecture can be configured. That is, the initial resolution result is not embedded as context information into the prompt words. Instead, the initial resolution result is used as the evaluation basis for the lineage resolution result. If a table name that exists in the initial resolution result is not found in the lineage resolution result, it indicates that there is a table name missing problem in the lineage resolution result. Based on this, the missing table name is fed back to update the prompt words.
[0042] The aforementioned feedback update means that the lineage resolution model parses the current SQL file only based on the prompt words that contain the aforementioned table names. When parsing other SQL files, it still uses the original prompt words that do not contain the aforementioned table names. After the current SQL file is parsed and passes the initial parsing result verification, that is, when it is determined that there is no missing table name problem in the lineage resolution result, the table name is removed from the prompt words.
[0043] Since the missing table name is temporarily updated in the prompt word, when the updated prompt word is input into the lineage resolution model again, the lineage resolution model will obviously focus on the missing table name when obtaining the lineage resolution result of the current SQL file based on the current prompt word, so as to ensure the acquisition of the data flow path related to the table name and further improve the completeness of the lineage resolution result.
[0044] The technical solution of this invention constructs prompt words based on the syntactic structure features of Structured Query Language (SCL). These prompt words include multiple operators, their associated parameters, and output formats. After obtaining the prompt words corresponding to the set of files to be parsed, the prompt words are input into a lineage parsing model. This allows the lineage parsing model to obtain the lineage parsing results for each SCL file based on the prompt words. By incorporating multiple SCL operators, their associated parameters, and output formats into the prompt words, the prompt words reflect the processing details of the data processing logic. This guides the lineage parsing model to perform deep parsing operations, avoiding insufficient parsing accuracy due to syntactic complexity and improving the data integrity of the lineage parsing results.
[0045] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant regions.
[0046] Example 2 Figure 2This is a flowchart of a data lineage analysis method provided in Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiments is that the output logic of the lineage analysis model is optimized through test sample iteration and prompt word iteration, such as... Figure 2 As shown, the method specifically includes: S201. Based on the prompt words, obtain the bloodline analysis results of the test sample set through the bloodline analysis model, and obtain the bloodline analysis deviation based on the bloodline analysis results of the test sample set and the expected bloodline analysis results.
[0047] First, a small number of representative test samples (e.g., less than or equal to 100) are used to perform batch inference on the output of the lineage resolution model under the current prompting strategy (i.e., prompt words). The purpose is to efficiently detect boundary conditions and typical resolution biases. The test samples can be drawn from different databases using stratified sampling of different types of test samples. At the same time, adversarial samples (e.g., samples that ignore specific operators) are injected to improve the robustness of the lineage resolution model test results.
[0048] By comparing the expected lineage resolution results with the actual lineage resolution results for the test sample set, we can identify the resolution deviations and their types. For example, resolution deviation types can include table-level deviations, field-level deviations, incomplete dependencies, and logical errors. Table-level deviations indicate omissions or errors in identifying the source or target table; field-level deviations indicate incorrect field sources; incomplete dependencies indicate the failure to recursively resolve the internal logic of views, stored procedures, or user-defined functions; and logical errors indicate misjudgments of data flow.
[0049] S202. Obtain a matching update strategy based on the bloodline analysis deviation, update the prompt words according to the update strategy, and increase the capacity of the test sample set.
[0050] Parsing deviations can include missing operators at the syntax level, in which case the matching update strategy can be to inject constraints, such as adding a specified new operator to the prompt; they can also include logical contradictions and redundancies caused by broken logic chains, in which case the matching update strategy can be to insert inference examples; and they can also include redundant output results, in which case the matching update strategy can be to strengthen concise instructions, such as adding a new output rule that specifies that the output result is completed within three inference steps.
[0051] Optionally, in this embodiment of the invention, obtaining the pedigree analysis deviation based on the pedigree analysis results and expected pedigree analysis results of the test sample set specifically includes: obtaining the analysis deviation type through a rule engine and a discriminant model based on the pedigree analysis results and expected pedigree analysis results of the test sample set. The rule engine is used to identify known analysis deviation types, such as detecting missing fields and format errors, which can be quickly identified using regular expressions or abstract syntax trees; the discriminant model is a lightweight artificial intelligence model that acts as a classifier, automatically clustering error categories to explore and obtain unknown analysis deviation types.
[0052] The parsing deviation types obtained through the rule engine and discriminant model can be summarized into a structured parsing report, such as "40% of the parsing deviations are due to unprocessed asynchronous calls" and "20% of the parsing deviations are due to ambiguity caused by lack of examples". Thus, the dual-track analysis method of rule engine and discriminant model can not only achieve the rapid acquisition of known deviation types, but also achieve the exploration and acquisition of unknown deviation types, ensuring the complete acquisition of various parsing deviation types in the lineage analysis results.
[0053] S203. Return to the execution of the operation of obtaining the bloodline analysis results of the test sample set through the bloodline analysis model according to the prompt words, and obtaining the bloodline analysis deviation according to the bloodline analysis results of the test sample set and the expected bloodline analysis results, until there is no bloodline analysis deviation in the bloodline analysis results of the test sample set.
[0054] After the prompt words are updated, the number of test samples in the test sample set is increased to verify the kinship analysis results of the kinship analysis model in the gradually expanding test sample set until there is no analysis bias in the kinship analysis results or the number of iterations reaches the preset threshold. At this point, the kinship analysis model training and testing can be considered complete.
[0055] The technical solution of this invention involves obtaining the kinship analysis results of a test sample set based on prompt words using a kinship analysis model, and obtaining the kinship analysis deviation based on the kinship analysis results of the test sample set and the expected kinship analysis results. A matching update strategy is then obtained based on the kinship analysis deviation to update the prompt words according to the update strategy. The test sample set is then expanded, and the kinship analysis results are iteratively obtained until there is no kinship analysis deviation in the kinship analysis results of the test sample set. Thus, by continuously updating the prompt words and increasing the number of test samples, the analysis accuracy of the kinship analysis model is continuously improved during the iteration process.
[0056] Example 3 Figure 3This is a flowchart of a data lineage analysis method provided in Embodiment 3 of the present invention. The relationship between this embodiment and the above embodiments is that, through a dual optimization strategy of verification instructions and confidence scores, the output fluctuation and illusion problems of the lineage analysis model during batch processing are solved. Figure 3 As shown, the method specifically includes: S301. Obtain the prompt words corresponding to the set of files to be parsed; wherein, the set of files includes multiple structured query language files; the prompt words are constructed based on the syntactic structure features of the structured query language; the prompt words include multiple operators and the associated parameters and output format of each operator.
[0057] S302. Add the verification instruction based on the abstract syntax tree to the prompt word, and input the prompt word into the lineage resolution model to obtain multiple lineage resolution results of each of the structured query language files through the lineage resolution model.
[0058] Embedding validation instructions based on abstract syntax trees in the prompt words can force the verification of the SQL syntax of the lineage resolution results, thereby filtering out obvious abnormal output results. For example, validation instructions can include requirements such as the FROM clause must be included, JOIN must explicitly specify the type, and undefined aliases are not allowed. By embedding these instructions, the output of the lineage resolution model is forced to be parsed through the abstract syntax tree, thus solving the illusion problem of the lineage resolution model's output results (e.g., generating invalid SQL queries). The above validation rules can quickly intercept logical errors (e.g., missing semicolons or invalid keywords).
[0059] S303. Among the multiple kinship analysis results, the target kinship analysis result with the highest confidence score shall be taken as the actual kinship analysis result.
[0060] For a set of files, multiple independent calls to the pedigree analysis model are initiated in parallel, keeping other parameters consistent and only fine-tuning the confidence interval of the model to obtain output results under different confidence intervals. Simultaneously, after each output result, the probability distribution returned by the pedigree analysis model is extracted, and the confidence score of the overall sequence is calculated based on this distribution. After sorting the confidence scores, the target pedigree analysis result with the highest confidence score is taken as the actual pedigree analysis result of the pedigree analysis model. This optimal selection method avoids the randomness of the output results, solves the output fluctuation problem of the pedigree analysis model, and also solves the problem of different output results from the same input during batch processing.
[0061] The technical solution of this invention incorporates verification instructions based on an abstract syntax tree into prompt words, and inputs these prompt words into a lineage resolution model. This model then obtains multiple lineage resolution results for each structured query language file. Among these results, the target lineage resolution result with the highest confidence score is selected as the actual lineage resolution result. Thus, by adding verification instructions to the prompt words, the illusion problem of the lineage resolution model's output results is solved. Simultaneously, the randomness of the output results is avoided through a selective optimization method, resolving the output fluctuation problem of the lineage resolution model.
[0062] Example 4 Figure 4 This is a structural block diagram of a data lineage analysis device provided in Embodiment 4 of the present invention. The device specifically includes: The prompt word acquisition module 401 is used to acquire prompt words corresponding to the set of files to be parsed; wherein, the set of files includes multiple structured query language files; the prompt words are constructed based on the syntactic structure features of structured query language; the prompt words include multiple operators and the associated parameters and output format of each operator; The parsing execution module 402 is used to input the prompt words into the lineage parsing model so that the lineage parsing model can obtain the lineage parsing results of each of the structured query language files based on the prompt words.
[0063] The technical solution of this invention constructs prompt words based on the syntactic structure features of Structured Query Language (SCL). These prompt words include multiple operators, their associated parameters, and output formats. After obtaining the prompt words corresponding to the set of files to be parsed, the prompt words are input into a lineage parsing model. This allows the lineage parsing model to obtain the lineage parsing results for each SCL file based on the prompt words. By incorporating multiple SCL operators, their associated parameters, and output formats into the prompt words, the prompt words reflect the processing details of the data processing logic. This guides the lineage parsing model to perform deep parsing operations, avoiding insufficient parsing accuracy due to syntactic complexity and improving the data integrity of the lineage parsing results.
[0064] Optionally, the parsing execution module 402 is specifically used to obtain the abstract syntax tree of the current structured query language file, and obtain the initial parsing result of the current structured query language file according to the abstract syntax tree; embed the initial parsing result as context information into the prompt word; and input the prompt word into the lineage parsing model so that the lineage parsing model obtains the lineage parsing result of each structured query language file based on the prompt word.
[0065] Optionally, the parsing execution module 402 is further configured to compare the lineage parsing result of the current structured query language file with the initial parsing result, and when it is determined that there is a missing table name in the lineage parsing result, to update the missing table name to the prompt word, and return to execute the operation of inputting the prompt word into the lineage parsing model for processing, so as to output the updated lineage parsing result.
[0066] Optionally, the data lineage analysis device is further configured to: obtain the lineage analysis results of the test sample set according to the prompt word through the lineage analysis model; obtain the lineage analysis deviation according to the lineage analysis results of the test sample set and the expected lineage analysis results; obtain a matching update strategy according to the lineage analysis deviation, update the prompt word according to the update strategy, and perform capacity expansion processing on the test sample set; return to execute the operations of obtaining the lineage analysis results of the test sample set according to the prompt word through the lineage analysis model, and obtaining the lineage analysis deviation according to the lineage analysis results of the test sample set and the expected lineage analysis results, until the lineage analysis results of the test sample set no longer have lineage analysis deviation.
[0067] Optionally, the data lineage analysis device is also used to obtain the lineage analysis deviation through a rule engine and a discrimination model based on the lineage analysis results and expected lineage analysis results of the test sample set.
[0068] Optionally, the parsing execution module 402 is further configured to add the verification instructions based on the abstract syntax tree to the prompt words, and input the prompt words into the lineage parsing model, so as to obtain multiple lineage parsing results for each of the structured query language files through the lineage parsing model; among the multiple lineage parsing results, the target lineage parsing result with the highest confidence score is taken as the actual lineage parsing result.
[0069] The above-described apparatus can execute the data lineage analysis method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the data lineage analysis method provided in any embodiment of the present invention.
[0070] Example 5 Figure 5A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0071] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0072] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0073] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data lineage resolution methods.
[0074] In some embodiments, the data lineage resolution method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on a heterogeneous hardware accelerator via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the data lineage resolution method described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform the data lineage resolution method by any other suitable means (e.g., by means of firmware).
[0075] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0076] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0077] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0078] To provide user interaction, the systems and techniques described herein can be implemented on a heterogeneous hardware accelerator, which includes: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the heterogeneous hardware accelerator. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including sound input, voice input, or haptic input).
[0079] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0080] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0081] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0082] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A method for data lineage analysis, characterized in that, include: Obtain the prompt words corresponding to the set of files to be parsed; wherein, the set of files includes multiple Structured Query Language (SCL) files; the prompt words are constructed based on the syntactic structure features of SCL; the prompt words include multiple operators and the associated parameters and output format of each operator; The prompt words are input into the lineage resolution model so that the lineage resolution model can obtain the lineage resolution results of each of the structured query language files based on the prompt words.
2. The data lineage analysis method according to claim 1, characterized in that, The step of inputting the prompt words into the lineage resolution model, so that the lineage resolution model obtains the lineage resolution results of each of the structured query language files based on the prompt words, includes: Obtain the abstract syntax tree of the current structured query language file, and obtain the initial parsing result of the current structured query language file based on the abstract syntax tree; The initial parsing result is embedded as context information into the prompt word; The prompt words are input into the lineage resolution model so that the lineage resolution model can obtain the lineage resolution results of each of the structured query language files based on the prompt words.
3. The data lineage analysis method according to claim 2, characterized in that, After obtaining the initial parsing result of the current structured query language file based on the abstract syntax tree, the process also includes: The lineage parsing result of the current structured query language file is compared with the initial parsing result. If it is determined that there is a missing table name in the lineage parsing result, the missing table name is fed back to update the prompt word, and the operation of inputting the prompt word into the lineage parsing model for processing is returned to output the updated lineage parsing result.
4. The data lineage analysis method according to claim 1, characterized in that, Also includes: Based on the prompt words, the bloodline analysis results of the test sample set are obtained through the bloodline analysis model, and the bloodline analysis deviation is obtained based on the bloodline analysis results of the test sample set and the expected bloodline analysis results. Based on the bloodline analysis deviation, a matching update strategy is obtained to update the prompt words according to the update strategy, and the test sample set is enlarged. Return to the execution of the operation of obtaining the bloodline analysis results of the test sample set through the bloodline analysis model according to the prompt words, and obtaining the bloodline analysis deviation according to the bloodline analysis results of the test sample set and the expected bloodline analysis results, until there is no bloodline analysis deviation in the bloodline analysis results of the test sample set.
5. The data lineage analysis method according to claim 4, characterized in that, The step of obtaining the bloodline analysis deviation based on the bloodline analysis results and the expected bloodline analysis results of the test sample set specifically includes: Based on the kinship analysis results and expected kinship analysis results of the test sample set, the kinship analysis deviation is obtained through the rule engine and the discrimination model.
6. The data lineage analysis method according to claim 1, characterized in that, The step of inputting the prompt words into the pedigree analysis model, so that the pedigree analysis model obtains the pedigree analysis results of each of the structured query language files based on the prompt words, specifically includes: The validation instructions based on the abstract syntax tree are added to the prompt words, and the prompt words are input into the lineage resolution model to obtain multiple lineage resolution results for each of the structured query language files through the lineage resolution model; Among the multiple kinship analysis results, the target kinship analysis result with the highest confidence score is taken as the actual kinship analysis result.
7. A data lineage analysis device, characterized in that, include: The prompt word acquisition module is used to acquire prompt words corresponding to the set of files to be parsed; wherein, the set of files includes multiple structured query language files; the prompt words are constructed based on the syntactic structure features of structured query language; the prompt words include multiple operators and the associated parameters and output format of each operator; The parsing and execution module is used to input the prompt words into the lineage parsing model, so that the lineage parsing model can obtain the lineage parsing results of each of the structured query language files based on the prompt words.
8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data lineage analysis method according to any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data lineage resolution method according to any one of claims 1-6.
10. A computer program product comprising a computer program that, when executed by a processor, implements the data lineage resolution method according to any one of claims 1-6.