A data blood relationship determination method and device, electronic equipment and storage medium

Abstract

The application discloses a data blood relationship determination method and device, electronic equipment and storage medium, and relates to the technical field of data processing. The method comprises the following steps: determining target workflow configuration information of to-be-processed data, wherein the target workflow configuration information comprises a target relational database and a target structured query language script of the to-be-processed data; processing the target relational database by using a pre-set language analysis rule to obtain an intermediate processing file of the to-be-processed data; and determining a data blood relationship of the to-be-processed data based on the intermediate processing file and the target structured query language script. According to the language analysis rule and the database type of the target relational database, the application determines an analysis editing file of the to-be-processed data, and then uses the target structured query language script to call the analysis editing file to analyze the data blood relationship of the to-be-processed data. Therefore, the efficiency and economy of the blood relationship analysis work are high without the aid of manual or semi-automatic tools.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for determining data lineage. Background Technology

[0002] Data lineage refers to the chain of relationships between data sources, indicating the upstream and downstream dependencies in data generation, transfer, and processing. It plays a crucial role in data quality, reliability, and interpretability. With the advent of the big data era and the increasing volume, complexity, and diversity of data, the performance of data lineage determination methods has become a major concern for data managers.

[0003] Currently, data lineage is determined based on data similarity. For example, data similarity is calculated based on the table names and structure information of each data point, and then a lineage score is determined based on the data similarity, thus obtaining the data lineage between the data. However, this method requires manual or semi-automated tools to label the data table names and structure information, and then determines the data lineage based on the labeled information. The analysis cycle for data lineage is long, costly, and has low practicality. Moreover, the larger the data volume, the slower the rate of determining data lineage. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for determining data lineage. It eliminates the need for manual or semi-automatic tools. Based on language parsing rules and the database type of the target relational database, it determines the parsing and editing file for the data to be processed, and then uses a target structured query language script to call the parsing and editing file to parse the data lineage of the data to be processed. This method is highly efficient and economical in lineage analysis.

[0005] According to one aspect of the present invention, a method for determining data lineage is provided, the method comprising:

[0006] Determine the target workflow configuration information for the data to be processed, including the target relational database and the target structured query language script for the data to be processed;

[0007] The target relational database is processed using pre-set language parsing rules to obtain intermediate processing files of the data to be processed;

[0008] Based on intermediate processing files and target structured query language scripts, the data lineage of the data to be processed is determined.

[0009] According to another aspect of the present invention, an apparatus for determining data lineage is provided. This apparatus is used to implement the data lineage determination method in any embodiment of the present invention. The apparatus includes:

[0010] The information acquisition module is used to determine the target workflow configuration information of the data to be processed, wherein the target workflow configuration information includes the target relational database and the target structured query language script of the data to be processed;

[0011] The file determination module is used to process the target relational database using pre-set language parsing rules to obtain intermediate processing files of the data to be processed.

[0012] The lineage determination module is used to determine the data lineage relationship of the data to be processed based on intermediate processing files and target structured query language scripts.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and a memory communicatively connected to the at least one processor;

[0015] The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to perform the method for determining data lineage in any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided that stores computer instructions for causing a processor to execute and implement the method for determining data lineage in any embodiment of the present invention.

[0017] The method for determining data lineage of the present invention includes: determining the target workflow configuration information of the data to be processed, wherein the target workflow configuration information includes the target relational database and the target structured query language script; processing the target relational database using pre-set language parsing rules to obtain an intermediate processing file of the data to be processed; and determining the data lineage of the data to be processed based on the intermediate processing file and the target structured query language script. The technical solution of the present invention first determines the parsing and editing file of the data to be processed according to the pre-set language parsing rules and the database type of the target relational database, and then uses the target structured query language script to call the parsing and editing file to parse out the data lineage of the data to be processed. The determination of data lineage is fully automatic, and the efficiency of lineage analysis is high. It does not require manual or semi-automatic tools, effectively reducing the cost of data lineage analysis and the impact of data volume on the determination of data lineage, thus improving the economy of data lineage determination. This paper addresses the problems of long analysis cycles, high costs, low practicality, and slower determination of data lineage relationships when manually or semi-automatically annotating data table descriptions and structure information and determining data lineage relationships based on the annotated information. Furthermore, the larger the data volume, the slower the determination of data lineage relationships becomes.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a method for determining data lineage provided by the present invention;

[0021] Figure 2 This is a schematic diagram of the structure of a blood relationship processing system provided by the present invention;

[0022] Figure 3 This is a schematic diagram of the structure of a data lineage determination device provided by the present invention;

[0023] Figure 4 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] Figure 1 This is a flowchart illustrating a method for determining data lineage provided by the present invention. This embodiment is applicable to situations such as automatically and efficiently constructing a data lineage database and visually querying data lineage. The method can be executed by the data lineage determination device provided by the present invention. This device can be implemented in hardware and / or software. In a specific embodiment, the device can be integrated into an electronic device. The following embodiments will illustrate this using the integration of the device into an electronic device as an example. Figure 1 The method specifically includes the following steps:

[0027] S101. Determine the target workflow configuration information for the data to be processed.

[0028] The data to be processed can be understood as data whose lineage needs to be determined. This could be newly generated data from the enterprise, or data whose lineage needs to be clarified by the user. This embodiment periodically acquires the enterprise's operational data, analyzes the generation, transfer, and processing of this operational data, obtains the lineage relationships of the operational data, and adds it to the data lineage database. The data in the data lineage database is regularly updated and enriched to comprehensively and meticulously address users' data lineage query tasks. The target workflow configuration information can be understood as the attribute information of the data to be processed, used to indicate the type and processing method of the data, including the target relational database and the target structured query language script for the data to be processed.

[0029] Specifically, the target relational database can be understood as the type, identifier, or name of the relational database containing the data to be processed, and the target structured query language script can be understood as the task processing or task execution logic for the data to be processed.

[0030] The advantage of this setup is that it allows us to determine the parsing and compilation files and parsing methods for the data to be processed based on the target workflow configuration information, thus enabling us to quickly determine the data lineage of the data to be processed.

[0031] Furthermore, the data to be processed includes at least one sub-data to be processed, and the target workflow configuration information includes at least one sub-target workflow configuration information. The sub-target workflow configuration information includes a sub-target relational database and a sub-target structured query language script. The sub-target workflow configuration information and the sub-data to be processed correspond one-to-one. In this embodiment, at least one sub-target workflow configuration information will also be stored in the data lineage analysis database (dp_lineage_process) to be processed so as to process the sub-data to be processed corresponding to each sub-target workflow configuration information in sequence.

[0032] This embodiment can configure task periodic scheduling rules through the cron expression, with a default time interval (20 seconds, 30 seconds, 50 seconds, etc.) to periodically trigger data lineage analysis, obtain the configuration information of the top-ranked sub-target workflow in the data lineage analysis database, and perform data lineage analysis on it.

[0033] In one implementation, S101 may specifically include: receiving an initial relational database and an initial structured query language script input by a user based on an interactive interface; determining whether a workflow generation instruction has been received; if a workflow generation instruction has been received, processing the initial relational database and the initial structured query language script using pre-set reference structure configuration information to obtain a target relational database and a target structured query language script.

[0034] The interactive interface can be understood as the interface through which information is exchanged with the user, including displays, touch screens, etc. The initial relational database can be understood as the database information input by the user that has not undergone standardization and cannot be accurately recognized by the lineage analysis processor. The initial structured query language script can be understood as the task execution script input by the user that has not undergone standardization and cannot be accurately recognized by the lineage analysis processor. The workflow generation instruction can be understood as the indication information triggered by the user to indicate that the workflow configuration information has been entered. The pre-set reference structure configuration information can be understood as a set of transformation information including the initial relational database and the initial structured query language script in a standardized manner. The target relational database can be understood as the initial relational database information that has undergone standardization and can be accurately recognized by the lineage analysis processor. The initial structured query language script can be understood as the initial structured query language script information that has undergone standardization and can be accurately recognized by the lineage analysis processor.

[0035] After the user inputs the initial relational database and initial structured query language script, they need to be converted into information that the processor (i.e., the lineage analysis processor) can recognize. This conversion requires a workflow generation command to initiate, which can be a workflow configuration save operation. Specifically, this embodiment can utilize a workflow configuration hook event mechanism to listen for workflow configuration save operations. Once a workflow configuration save operation is detected, the initial relational database and initial structured query language script are converted into target relational database and target structured query language script by referring to the structure configuration information.

[0036] For example, users can configure Structured Query Language (SQL) tasks by dragging, dropping, and scrambling, determine the initial relational database based on the database scope corresponding to the SQL task, and determine the initial structured query language script based on the execution method of the SQL task.

[0037] Optionally, if the data to be processed is newly generated operational data of the enterprise acquired periodically, the processor can also automatically create a structured query language (SQL) task for the operational data, determine the relational database of the SQL task and the SQL script for performing data lineage analysis. The relational database of the SQL task can be understood as the target relational database in this embodiment, and the SQL script can be understood as the target structured query language script in this embodiment.

[0038] S102. Process the target relational database using pre-set language parsing rules to obtain intermediate processing files of the data to be processed.

[0039] In this embodiment, the tool used to determine language parsing rules is an open-source parser called ANTLR (Another Tool for Language Recognition), which automatically generates and visualizes a syntax tree based on the input. The language parsing rules can be understood as pre-written g4 files containing syntax rules for various types of relational databases, and the intermediate processing files can be understood as package files containing target structured query language scripts used to process the data to be processed.

[0040] Specifically, a g4 file is a pre-written file that performs lexical and syntactic analysis on SQL scripts for different databases. It is primarily related to the relational database type; different relational databases require different g4 files. For example, a mysql.g4 file is needed for MySQL, and a postgreSQL.g4 file is needed for PostgreSQL. To save data processing time, this embodiment completes the writing of the g4 file before data processing. For instance, in the gene sequencing example, the g4 file is similar to a gene sequencing standard for pre-sequencing gene sequences. The g4 file allows for lexical and syntactic analysis of the database, essentially sequencing genes using the gene sequencing standard.

[0041] In one implementation, S102 may specifically include: determining the database type of the target relational database; searching in the language parsing rules based on the database type to obtain a syntax description file of the data to be processed; and performing syntax compilation processing on the language description file to obtain an intermediate processing file.

[0042] Based on the above description, determining the database type of the target relational database can be understood as determining whether the database is a MySQL database or a PostgreSQL database. Finding the syntax description file of the data to be processed by searching the language parsing rules based on the database type can be understood as searching the g4 file based on the database type to determine the syntax description file. When the database is a MySQL database, the syntax description file is mysql.g4 file, and when the database is a PostgreSQL database, the syntax description file is postgreSQL.g4 file.

[0043] Assuming the intermediate processing file is a JAR file and the target relational database is MySQL, the intermediate processing file is obtained by syntactically compiling the language description file. This includes: generating lexical and syntax analysis code from the command line (e.g., antlr4-Dlanguage = Java mysql.g4), compiling the Java code generated by ANTLR (javac*.java), and forming the mysql-parser.jar file.

[0044] Command line input refers to generating lexical and syntax analyzer code through the anltr command line tool. For example, to generate lexical and syntax analyzer code for a MySQL database in Java, execute the command antlr4-Dlanguage=Javamysql.g4. It is worth noting that this command can be adjusted and modified according to different programming languages ​​and different databases, but this embodiment does not limit it.

[0045] S103. Based on the intermediate processing file and the target structured query language script, determine the data lineage of the data to be processed.

[0046] In this context, data lineage can be understood as the node information and hierarchical dependencies in the processes of data generation, transfer, use, and processing. Specifically, a data lineage includes at least one sub-data lineage, with each sub-data lineage corresponding one-to-one with a sub-data item to be processed.

[0047] In one implementation, S103 may specifically include: determining the syntax logic tree of the data to be processed based on the intermediate processing file and the target structured query language script; and determining the data lineage based on the semantic analysis results of the syntax logic tree, wherein the data lineage includes at least one of the data source table association relationship and field association relationship of the data to be processed.

[0048] The syntax logic tree can be understood as a syntax logic relationship generated by calling intermediate processing files using the target structured query language script for syntax analysis.

[0049] Specifically, assuming the intermediate processing file is mysql-parser.jar and the target structured query language script is an SQL script, determining the data lineage of the data to be processed based on the intermediate processing file and the target structured query language script can include: calling the intermediate processing file (mysql-parser.jar) based on the target structured query language script (the determined SQL script) to perform lexical analysis to generate an Abstract Syntax Tree (AST); using the AST to perform semantic analysis of the data lineage analysis logic to form the association relationship of the data source tables or fields, thereby obtaining the node information and hierarchical dependencies of the generation, transfer, use, and processing of the data to be processed. It is worth noting that this embodiment also stores the data lineage of the data to be processed to handle user data lineage query and retrieval tasks. In such cases, there is no need to reconfirm the data lineage; only the stored data lineage needs to be retrieved, reducing the workload of the processor.

[0050] Furthermore, the data lineage analysis results are stored in the data lineage database (dp_lineage_result) in a standard format such as nodes and / or edges. Generally, Neo4j graph database is used for storage by default. Graph databases can provide efficient data management and query analysis capabilities. The advantage of this setting is that it can quickly process complex relationship queries.

[0051] Optionally, when determining data lineage based on the semantic analysis results of the syntax logic tree, the method may also include: determining whether the recognition accuracy of the data source table association and / or field association is less than a preset accuracy threshold; if the recognition accuracy is less than the preset accuracy threshold, then the recognition of the data source table association and / or field association is identified as an accuracy anomaly.

[0052] The preset accuracy threshold represents the accuracy of data recognition. The specific parameter value is related to the correlation index requirements of data lineage. The preset accuracy threshold ranges from 0 to 1, and this embodiment does not limit it.

[0053] For example, assuming a preset accuracy threshold of 0.85, the metadata extraction scope parameter is adjusted to Schema, Table, and Column (Metadata Extraction Scope parameter range is Schema, Table, Column). The accuracy of identifying the relationships between data source tables and / or field relationships is then determined to be no less than 0.85. If the accuracy is no less than 0.85, the data identification is successful; otherwise, it is unsuccessful. Furthermore, unsuccessfully identified data needs to be marked with an exception to alert the user to the data anomaly and allow the user to add data lineage information. Secondly, if the syntax involves multiple table objects with JOIN operations, they will be identified as needing to be merged. The merged result will form guiding data for constructing a data lineage diagram.

[0054] Optionally, after determining the data lineage of the data to be processed based on the intermediate processing file and the target structured query language script, the method further includes: when a lineage query instruction sent by the user is received, matching is performed in at least one sub-data lineage based on the lineage query instruction, and candidate data lineage is determined according to the matching results; and the interactive interface is controlled to display the candidate data lineage.

[0055] The lineage query command can be understood as the lineage relationships that the user needs to understand and view. For example, the relationship between tables A1 and B2 in a local housing and construction department's database DB1 and table B1 in a natural resources department's database DB2; the full lifecycle lineage relationships of table A1 in a local housing and construction department's database DB1; and the full lifecycle lineage relationships of table B1 in a local natural resources department's database DB2. Candidate data lineage relationships can be understood as the associations stored in the data lineage database that match the lineage query command. Based on the lineage query command, matching is performed on at least one sub-data lineage relationship, and candidate data lineage relationships are determined based on the matching results. Upon receiving a lineage query command, the data lineage analysis program is invoked to retrieve the structured data corresponding to the lineage query command from the data lineage database (dp_lineage_process) for analysis, forming the associations between database tables and / or fields, i.e., candidate data lineage relationships. Displaying candidate data lineage relationships on the interactive interface can be understood as visually displaying the queried database tables and / or fields to help users understand the data flow process and overall overview.

[0056] Furthermore, this embodiment also provides a linked display of metadata information, making it convenient for users to understand the structure and data volume of the data entity table. This data transparency not only facilitates data analysis and governance, but also enhances users' confidence in data quality and strengthens data credibility.

[0057] It is worth noting that this embodiment is based on the D3.js framework to design the data interaction interface. It can use a dynamic, draggable visual canvas to display the overall overview of the data lineage. Users can enter keywords to retrieve the corresponding data entities and display the associated metadata details, which greatly simplifies the user's data query operation and improves the user's data query experience.

[0058] Optionally, when matching in at least one sub-data lineage relationship based on the lineage relationship query command and determining candidate data lineage relationships based on the matching results, the method further includes: determining whether the response success rate of the data query interface meets the preset response conditions, wherein the data query interface is used to transmit at least one sub-data lineage relationship or candidate data lineage relationship; if the response success rate does not meet the preset response conditions, a response error message is generated and the interactive interface is controlled to display the response error message.

[0059] The data query interface can be understood as a data transmission interface. In this embodiment, the data query application programming interface (API) calls the data lineage analysis program to obtain the structured data corresponding to the lineage query command from the data lineage database (dp_lineage_process). The call includes transmitting all data first and then parsing, and parsing first and then transmitting the parsed data. Therefore, the data query interface can transmit at least one sub-data lineage relationship or candidate data lineage relationship. The specific transmission content is related to the data call logic.

[0060] The preset response conditions can be understood as the basis for measuring the stability of the data lineage query operation. For example, the response success rate of the data query interface is not less than 90%, the response success rate of the data query interface is not less than 85%, or the data change difference index is less than 0.1. The corresponding exception prompt can be understood as a query exception prompt message, used to inform the user that there is an anomaly in the query operation, so that the user can deal with the fault in time and ensure the stability of the data lineage query system. Specifically, assuming the preset response conditions are that the response success rate of the data query interface is not less than 90% or the data change difference index is less than 0.1, once it is determined that the response success rate of the data query interface is less than 90% or the data change difference index is not less than 0.1, a response exception prompt is generated, and the interactive interface is controlled to display the response exception prompt, notifying the user to take timely measures to deal with the system exception.

[0061] It is worth noting that the monitoring work is triggered on a timed basis. For example, a sampling frequency of 30 minutes can be set to periodically collect the success rate of the data lineage interface response. The advantage of this setting is that it can continuously monitor the performance indicators, task dependencies and metadata changes of data lineage analysis, promptly identify and resolve potential problems in the data processing process, and ensure the efficiency of data processing and the stability of data quality.

[0062] For example, if the lineage query command is for the relationship between tables A1 and B2 in a local housing and construction department's database DB1 and table B1 in a natural resources department's database DB2, then the data lineage relationship is the node association relationship between tables A1 and A2 in the housing and construction department's DB1 and table B1 in the natural resources department's DB2. The interactive interface will display the node association relationship between tables A1, A2, and table B1 in the natural resources department's DB2, as well as detailed metadata information for the nodes in tables A1, A2, and B1. This associated metadata helps users understand the structure and volume of data in the database tables, enabling them to quickly locate and resolve data problems and ensure data accuracy and reliability.

[0063] Figure 2 This is a schematic diagram of the structure of a blood relationship processing system provided by the present invention. Figure 2 As can be seen from the data, the system consists of four parts: front-end, service, analysis, and monitoring. The front-end refers to the system front-end, which includes the data lineage dashboard module and the workflow configuration module. The service refers to the open service, which includes the data lineage service module and the metadata module. The analysis refers to the data lineage analysis, which includes the workflow scheduling module and the data lineage analysis module. The monitoring refers to the data lineage monitoring module.

[0064] This embodiment's workflow configuration module supports task types such as SQL, Shell, and Python. It primarily uses a Directed Acyclic Graph (DAG) approach for task orchestration. When configuring SQL tasks, it supports database script configuration for various relational database types, including MySQL, PostgreSQL, Oracle, and SQL Server. Common workflow engines include Airflow, DolphinScheduler, and Azkaban. This embodiment's workflow scheduling module uses DolphinScheduler as the underlying task scheduling engine. This engine enables workflow task scheduling configuration, task instance execution, and real-time task status monitoring. It also automatically triggers the data lineage analysis module to perform automated data lineage analysis on the task relationships defined in the workflow and the SQL task script content. The data lineage analysis module uses the ANTLR syntax analysis framework to perform lexical analysis, syntax analysis, and parse tree traversal on the input SQL script. Then, following the logic of data lineage analysis, it parses out information such as data tables, fields, and relationships, storing this information in a data lineage database for providing data lineage query services. The Data Lineage Service module provides interface services such as data lineage query and report query. The Data Lineage Dashboard module enables interactive visual information display, showing the complete data flow path, including the data's source, direction, processing tasks, and destination. This typically includes nodes and edges; nodes represent data entities or processing tasks (e.g., database tables, files, interfaces), and edges represent the data flow direction. Furthermore, the Data Lineage Dashboard module supports search and filtering functions. For example, clicking a node displays the metadata information of the data entity (database table name, storage size, storage location, etc.), and also supports drag-and-drop, zooming, and lineage highlighting. The Data Lineage Monitoring module monitors and tracks the execution status of data lineage analysis tasks, including the correctness of the analysis, execution time, and success status, helping operations personnel to promptly identify and resolve analysis failures. Metadata describes data attributes, typically describing the data's name, type, storage location, data size, maintainer, etc., reflecting certain characteristics of the data. The metadata module provides metadata information query and retrieval for database tables and fields, supporting the function of locating database tables to associated data lineage dashboards.

[0065] This embodiment supports the use of data lineage data by different applications, including alarms for changes in data lineage, and supports the association of metadata modules with the database table objects of data lineage, providing rich data detail display.

[0066] The technical solution of this embodiment first determines the parsing and editing file of the data to be processed based on pre-set language parsing rules and the database type of the target relational database. Then, the target structured query language script calls the parsing and editing file to parse out the data lineage of the data to be processed. The determination of data lineage is fully automatic, and the efficiency of lineage analysis is high. It does not require manual or semi-automatic tools, which can effectively reduce the cost of data lineage analysis and the impact of data volume on the determination of data lineage, thus improving the economy of data lineage determination. It solves the problems of long analysis cycles, high costs, low practicality, and the decreasing speed of data lineage determination as the data volume increases, which are caused by manually or semi-automatically annotating the data table labels and table structure information and determining data lineage based on the annotated information.

[0067] Figure 3 This is a schematic diagram of the structure of a data lineage determination device provided by the present invention. Figure 3 As shown, the device includes: an information acquisition module 301, a document determination module 302, and a bloodline determination module 303.

[0068] The information acquisition module 301 is used to determine the target workflow configuration information of the data to be processed, wherein the target workflow configuration information includes the target relational database and the target structured query language script of the data to be processed.

[0069] The file determination module 302 is used to process the target relational database using pre-set language parsing rules to obtain intermediate processing files of the data to be processed.

[0070] The lineage determination module 303 is used to determine the data lineage relationship of the data to be processed based on the intermediate processing file and the target structured query language script.

[0071] Optionally, the information acquisition module 301 is specifically used to receive the initial relational database and initial structured query language script input by the user based on the interactive interface; determine whether a workflow generation instruction has been received; if a workflow generation instruction has been received, process the initial relational database and initial structured query language script using the pre-set reference structure configuration information to obtain the target relational database and target structured query language script.

[0072] Optionally, the file determination module 302 is specifically used to determine the database type of the target relational database; search in the language parsing rules based on the database type to obtain the syntax description file of the data to be processed; and perform syntax compilation processing on the language description file to obtain an intermediate processing file.

[0073] Optionally, the lineage determination module 303 is specifically used to determine the syntax logic tree of the data to be processed based on the intermediate processing file and the target structured query language script; and to determine the data lineage relationship based on the semantic analysis results of the syntax logic tree, wherein the data lineage relationship includes at least one of the data source table association relationship and field association relationship of the data to be processed.

[0074] Optionally, the lineage determination module 303 is also used to determine whether the recognition accuracy of the data source table association and / or field association is less than a preset accuracy threshold when determining the data lineage relationship based on the semantic analysis results of the syntax logic tree; if the recognition accuracy is less than the preset accuracy threshold, the recognition identifier of the data source table association and / or field association is determined to be an accuracy abnormality.

[0075] Optionally, the data to be processed includes at least one sub-data to be processed, and the data lineage includes at least one sub-data lineage, with each sub-data lineage corresponding one-to-one with a sub-data to be processed.

[0076] Optionally, the lineage determination module 303 is also used to determine the data lineage relationship of the data to be processed based on the intermediate processing file and the target structured query language script, and when a lineage relationship query instruction sent by the user is received, to match in at least one sub-data lineage relationship based on the lineage relationship query instruction, and to determine candidate data lineage relationships based on the matching results; and to control the interactive interface to display the candidate data lineage relationships.

[0077] Optionally, the lineage determination module 303 is further configured to determine whether the response success rate of the data query interface meets preset response conditions when matching in at least one sub-data lineage relationship based on the lineage relationship query instruction and determining candidate data lineage relationships based on the matching results. The data query interface is used to transmit at least one sub-data lineage relationship or candidate data lineage relationship. If the response success rate does not meet the preset response conditions, a response error message is generated, and the interactive interface is controlled to display the response error message.

[0078] The data lineage determination device provided in this embodiment of the invention can execute the data lineage determination method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0079] Figure 4This is a schematic diagram of the structure of an electronic device provided by the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0080] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the read-only memory (ROM) 12 or loaded from storage unit 18 into the random access memory (RAM) 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0081] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0082] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as methods for determining data lineage.

[0083] In some embodiments, the method for determining data lineage may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for determining data lineage described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the method for determining data lineage by any other suitable means (e.g., by means of firmware).

[0084] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0085] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0086] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0087] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0088] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0089] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0090] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0091] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining data lineage, characterized in that, include: Determine the target workflow configuration information for the data to be processed, wherein the target workflow configuration information includes the target relational database and the target structured query language script for the data to be processed; Determine the database type of the target relational database; Based on the database type, a search is performed in the language parsing rules to obtain the syntax description file of the data to be processed; The language description file is syntactically compiled to obtain an intermediate processing file; Based on the intermediate processing file and the target structured query language script, determine the syntax logic tree of the data to be processed; Based on the semantic analysis results of the syntax logic tree, the data lineage is determined, wherein the data lineage includes at least one of the data source table association relationship and field association relationship of the data to be processed; In determining data lineage based on the semantic analysis results of the syntax logic tree, the method further includes: Determine whether the accuracy of identifying data source table relationships and / or field relationships is less than a preset accuracy threshold; If the recognition accuracy is less than the preset accuracy threshold, the recognition of the data source table association and / or field association is identified as having an accuracy anomaly.

2. The method according to claim 1, characterized in that, The target workflow configuration information for determining the data to be processed includes: Receives initial relational database and initial structured query language script from user input via an interactive interface; Determine if a workflow generation instruction has been received; If the workflow generation instruction is received, the initial relational database and the initial structured query language script are processed using the pre-set reference structure configuration information to obtain the target relational database and the target structured query language script.

3. The method according to claim 1, characterized in that, The data to be processed includes at least one sub-data to be processed, and the data lineage includes at least one sub-data lineage, with each sub-data lineage corresponding one-to-one with the sub-data to be processed. After determining the data lineage of the data to be processed based on the intermediate processing file and the target structured query language script, the method further includes: When a bloodline query instruction is received from a user, the bloodline query instruction is used to match the bloodline relationships in at least one sub-data relationship, and candidate data relationships are determined based on the matching results. The control interface displays the lineage relationships of the candidate data.

4. The method according to claim 3, characterized in that, When matching the bloodline relationship query instruction in the at least one sub-data bloodline relationship and determining the candidate data bloodline relationship based on the matching result, the method further includes: Determine whether the response success rate of the data query interface meets the preset response conditions, wherein the data query interface is used to transmit the at least one sub-data lineage relationship or the candidate data lineage relationship; If the response success rate does not meet the preset response conditions, a response error message is generated, and the interactive interface is controlled to display the response error message.

5. A device for determining data lineage, characterized in that, The method for determining data lineage as described in any one of claims 1 to 4, wherein the device for determining data lineage comprises: The information acquisition module is used to determine the target workflow configuration information of the data to be processed, wherein the target workflow configuration information includes the target relational database and the target structured query language script of the data to be processed; The file determination module is used to determine the database type of the target relational database; search the language parsing rules based on the database type to obtain the syntax description file of the data to be processed; and perform syntax compilation processing on the language description file to obtain an intermediate processing file. The lineage determination module is used to determine the syntax logic tree of the data to be processed based on the intermediate processing file and the target structured query language script; and to determine the data lineage relationship based on the semantic analysis results of the syntax logic tree, wherein the data lineage relationship includes at least one of the data source table association relationship and field association relationship of the data to be processed; The lineage determination module is also used to determine whether the recognition accuracy of the data source table association and / or field association is less than a preset accuracy threshold; if the recognition accuracy is less than the preset accuracy threshold, the recognition identifier of the data source table association and / or field association is determined to be an accuracy anomaly.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the method for determining data lineage as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for determining data lineage as described in any one of claims 1 to 4.

Images