Data lineage generation method, device, equipment and medium with verification mechanism

Abstract

The present application relates to a data lineage generation method, device, equipment and medium with a verification mechanism. The method includes: obtaining task data to be classified and attribute information of the task data; parsing the attribute information to obtain a first data lineage result corresponding to the task data, and storing the first data lineage result as the lineage relationship of the task data in a database; a plurality of task data and their corresponding lineage relationships are stored in the database; the task data and attribute information are input into a lineage extraction model to obtain a second data lineage result corresponding to the task data; when the first data lineage result and the second data lineage result are inconsistent, the lineage relationship in the database is updated to the second data lineage result. This method can perform double verification on the lineage relationship of the task data, thereby improving the accuracy of the generated data lineage relationship.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and medium for generating data lineage with a verification mechanism. Background Art

[0002] Data lineage refers to the record of all the steps and relationships that data has gone through in its entire life cycle, from generation, storage, processing, conversion, dissemination to final use. This record is similar to a human family tree, which can track the source, flow, processing and change history of data.

[0003] In traditional technology, the common practice of generating data lineage is as follows: (1) Establish a unified metadata management platform: uniformly record and manage the definition, source, flow process and status of all data in different systems; (2) Implement standardized data governance strategies: formulate unified data naming, classification and flow rules to reduce errors caused by individual or team differences; (3) Use automated lineage capture tools: use automated tools to track data flow, reduce manual entry errors, and improve update efficiency; (4) Conduct regular lineage verification and auditing: regularly verify and audit the lineage relationships in the system to promptly detect and correct deviations.

[0004] However, in the current data lineage generation method, the accuracy of the generated data lineage relationship lacks sufficient data support. Summary of the invention

[0005] Based on this, it is necessary to provide a data lineage generation method, device, equipment and medium with a verification mechanism that can improve the accuracy of the generated data lineage relationship in response to the above technical problems.

[0006] In a first aspect, the present application provides a data lineage generation method with a verification mechanism, comprising:

[0007] Obtaining task data to be classified and attribute information of the task data;

[0008] Parsing the attribute information to obtain a first data lineage result corresponding to the task data, and storing the first data lineage result as the lineage relationship of the task data in a database; the database stores a plurality of task data and their corresponding lineage relationships;

[0009] Inputting the task data and attribute information into the blood relationship extraction model to obtain the second data blood relationship result corresponding to the task data;

[0010] When the first data lineage result and the second data lineage result are inconsistent, the lineage relationship in the database is updated to the second data lineage result.

[0011] In one embodiment, the blood relationship extraction model is obtained by the following method:

[0012] Extracting task data and its corresponding blood relationship in the database to form a first data set;

[0013] Acquire the instance data generated in the workflow to form a second data set;

[0014] The first data set and the second data set are used to train an initial blood relationship extraction model to adjust parameters of the initial blood relationship extraction model to obtain a trained blood relationship extraction model.

[0015] In one embodiment, the blood relationship extraction model is a K nearest neighbor model;

[0016] The initial blood relationship extraction model is trained using the first dataset and the second dataset, including:

[0017] Use the first data set as the initial sample set of the K nearest neighbor model, classify the instance data in the second data set, and obtain the blood relationship corresponding to the instance data;

[0018] The instance data and its corresponding blood relationship are merged into the initial sample set to obtain the predicted sample set of the K nearest neighbor model.

[0019] In one embodiment, adjusting the parameters of the initial blood relationship extraction model includes:

[0020] Adjust the prediction category of the K nearest neighbor model, adjust the distance measurement method of the K nearest neighbor model, and adjust the K value of the K nearest neighbor model.

[0021] In one embodiment, the attribute information is parsed to obtain the first data lineage result corresponding to the task data, including:

[0022] Parsing the attribute information to generate an abstract syntax tree corresponding to the attribute information; the abstract syntax tree is used to represent the lexical information, grammatical information and semantic information contained in the attribute information;

[0023] Each abstract syntax tree is divided into hierarchical nodes, and a first data lineage result corresponding to the task data is constructed according to the dependency relationship between multiple abstract syntax trees with dependencies.

[0024] In one embodiment, constructing a first data lineage result corresponding to the task data according to the dependency relationship between a plurality of abstract syntax trees having dependency relationships includes:

[0025] Split the abstract syntax tree to obtain multiple sub-statements of the abstract syntax tree;

[0026] Determine the data lineage relationship contained in the abstract syntax tree based on multiple sub-statements;

[0027] When the abstract syntax tree is split, the abstract syntax tree is parsed according to the data lineage relationship to obtain the lineage data of the structured query language script.

[0028] In a second aspect, the present application also provides a data lineage generation device with a verification mechanism, comprising:

[0029] An information acquisition module, used to acquire task data to be classified and attribute information of the task data;

[0030] The parsing module is used to parse the attribute information to obtain the first data lineage result corresponding to the task data, and store the first data lineage result as the lineage relationship of the task data in the database; the database stores a plurality of task data and their corresponding lineage relationships;

[0031] An extraction module, used for inputting the task data and attribute information into a blood relationship extraction model to obtain a second data blood relationship result corresponding to the task data;

[0032] The updating module is used to update the blood relationship in the database to the second data blood relationship result when the first data blood relationship result and the second data blood relationship result are inconsistent.

[0033] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods described in the first aspect of the present application are implemented.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the methods described in the first aspect of the present application.

[0035] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect of the present application.

[0036] The above-mentioned data lineage generation method, device, equipment and medium with a verification mechanism obtain the task data to be classified and the attribute information of the task data; parse the attribute information to obtain the first data lineage result corresponding to the task data, and store the first data lineage result as the lineage relationship of the task data in the database; input the task data and the attribute information into the lineage extraction model to obtain the second data lineage result corresponding to the task data; when the first data lineage result and the second data lineage result are inconsistent, the lineage relationship in the database is updated to the second data lineage result, thereby realizing the verification of the generated data lineage relationship and improving the accuracy of the generated data lineage relationship. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a flowchart of a method for generating data lineage with a verification mechanism in one embodiment;

[0039] Figure 2 A schematic diagram of a system application architecture for implementing a data lineage generation method with a verification mechanism provided by the present application in one embodiment;

[0040] Figure 3 It is a structural block diagram of a data lineage generation device with a verification mechanism in one embodiment;

[0041] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] The problem of data lineage accuracy involves the traceability, tracking and link visualization of data in the data management system. It is a key issue in the field of data management. It mainly includes the diversity of data sources, the complexity of data processing, the distribution of data storage, the extensiveness of data application, and the complexity of technical implementation. In general, the problem of data lineage accuracy is a multi-dimensional and multi-level challenge that involves all aspects of data management.

[0044] Specifically, the problem of data lineage accuracy involves many aspects: (1) Diversity of data sources: Data integration from different channels. Data comes from multiple channels such as APP, Web, counter, etc., and is finally imported into the database through various applications. When integrating data from different sources, their respective sources must be accurately recorded to ensure the accuracy of data lineage. (2) Complexity of data processing: Multi-stage processing flow. From data acquisition to final consumption, data will go through multiple processing stages. At each stage, data may be converted, cleaned and aggregated, and these operations must be reflected in the data lineage. A combination of multiple processing technologies, using different technologies such as SparkSQL and FlinkSQL for processing. Each technology may have different effects on the recording method and accuracy of data lineage; (3) Distribution of data storage: Cross-platform data storage. Data may be stored in multiple storage systems such as RDS, MQ, MPP, etc. The differences in storage structure of each system pose challenges to the automatic capture and management of data lineage. Heterogeneity of data formats. Different storage systems support different data formats, such as text, binary, database tables, etc. (4) Extensive data application: Data sharing across businesses. Data applications cover multiple aspects such as indicator systems, reporting systems, and data services. Each business has different requirements for data lineage, but all require accurate lineage information to maintain data traceability and consistency.

[0045] In traditional technology, in order to improve the accuracy of data lineage, the usual practices are as follows: (1) Establish a unified metadata management platform: uniformly record and manage the definition, source, flow process and status of all data in different systems; (2) Implement standardized data governance strategies: formulate unified data naming, classification and flow rules to reduce errors caused by individual or team differences; (3) Use automated lineage capture tools: use automated tools to track data flow, reduce manual entry errors, and improve update efficiency; (4) Conduct regular lineage verification and auditing: regularly verify and audit the lineage relationships in the system to promptly detect and correct deviations.

[0046] The starting point of the existing technology is to improve the accuracy of data lineage by building new platforms, adopting more efficient tools, and formulating processes and systems. However, the accuracy of data lineage lacks effective data support. The purpose of the present invention is to calculate and demonstrate the accuracy of data lineage from a large amount of sample data.

[0047] In one embodiment, Figure 1 As shown, a data lineage generation method with a verification mechanism is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0048] Step 102: Obtain task data to be classified and attribute information of the task data.

[0049] Among them, task data can be the data of work projects obtained on the big data platform, which is used to characterize the workflow or work tasks in the project, and is an abstract and general description of the business rules between the workflow and its various operation steps. Exemplarily, task data can also be called static data. Attribute information is used to characterize the relevant attributes of task data, such as task time, task node, task parameters, etc.

[0050] Optionally, the task scheduling platform can be used to manage and execute scheduled or planned tasks, trigger scheduled tasks, manage task dependencies, schedule tasks, and handle errors. The task execution platform can also be used to execute tasks and generate task status data as part of the task data.

[0051] Step 104, parse the attribute information to obtain the first data lineage result corresponding to the task data, and store the first data lineage result as the lineage relationship of the task data in the database; the database stores a plurality of task data and their corresponding lineage relationships.

[0052] Specifically, the first data lineage result can be obtained based on an existing data lineage analysis method, such as a syntax analysis method. For example, the attribute information can represent that data A is processed to generate data B, and then it can be analyzed that A and B have a lineage relationship, and data A is the upstream data of data B, and data B is the downstream data of data A.

[0053] Step 106, input the task data and attribute information into the blood relationship extraction model to obtain the second data blood relationship result corresponding to the task data.

[0054] Exemplarily, the blood relationship extraction model can be an algorithm model based on machine learning. Among them, machine learning is a multi-disciplinary cross-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. By studying how computers simulate or implement human learning behavior, new knowledge or skills can be acquired, and the existing knowledge structure can be reorganized to continuously improve its performance. In application practice, machine learning algorithm models can be used to classify and cluster various types of data. For the blood relationship extraction of data, the task data, attribute information and the corresponding data blood relationship in the existing data set can be used to train the blood relationship extraction model, so that the blood relationship extraction model can extract the corresponding data blood relationship according to the newly input task data and attribute information as the second data blood relationship result.

[0055] Step 108: When the first data lineage result and the second data lineage result are inconsistent, the lineage relationship in the database is updated to the second data lineage result.

[0056] In the embodiment of the present application, a double verification mechanism is adopted for the blood relationship of the working data. Double verification refers to verifying the accuracy of the data through two different methods or approaches to ensure the consistency and reliability of the data. This method is particularly important in data processing and analysis, and can reduce the errors and risks that may be caused by a single method.

[0057] Since the first data kinship result is obtained based on traditional methods such as grammatical analysis, its accuracy lacks effective data support. The second data kinship result is obtained based on a trained kinship extraction model, which can be trained using a large amount of sample data to improve its classification accuracy.

[0058] Therefore, when the first data bloodline result and the second data bloodline result are inconsistent, the bloodline relationship in the database is updated to the second data bloodline result, thereby improving the accuracy of the bloodline relationship generation result through the double verification mechanism.

[0059] In the above-mentioned data lineage generation method with a verification mechanism, a double verification mechanism is adopted, which is achieved by obtaining the task data to be classified and the attribute information of the task data; parsing the attribute information to obtain the first data lineage result corresponding to the task data, and storing the first data lineage result as the lineage relationship of the task data in the database; inputting the task data and the attribute information into a lineage relationship extraction model to obtain the second data lineage result corresponding to the task data; when the first data lineage result and the second data lineage result are inconsistent, the lineage relationship in the database is updated to the second data lineage result, thereby realizing the verification of the generated data lineage relationship and improving the accuracy of the generated data lineage relationship.

[0060] In one embodiment, the blood relationship extraction model is obtained by the following method:

[0061] Step 202, extracting task data and its corresponding blood relationship in the database to form a first data set;

[0062] Step 204, acquiring instance data generated in the workflow to form a second data set;

[0063] Step 206: Use the first data set and the second data set to train the initial blood relationship extraction model to adjust the parameters of the initial blood relationship extraction model to obtain a trained blood relationship extraction model.

[0064] Specifically, the machine learning model can be trained through the data set to adjust the model parameters so that the model can accurately classify the data. Among them, the task data and its corresponding blood relationship in the data can be used as training data and labels to constitute the first data set for preliminary training of the blood relationship extraction model. The instance data generated in the workflow can constitute the second data set due to the large sample size, which is used for further fine-tuning the blood relationship extraction model. By training the blood relationship extraction model with a large amount of instance data, the classification accuracy of the blood relationship extraction model can be improved.

[0065] In an exemplary embodiment, the blood relationship extraction model is a K nearest neighbor model. Using the first data set and the second data set to train the initial blood relationship extraction model includes:

[0066] Step 302, using the first data set as an initial sample set of the K nearest neighbor model, classifying the instance data in the second data set, and obtaining the blood relationship corresponding to the instance data;

[0067] Step 304: merge the instance data and its corresponding blood relationship into the initial sample set to obtain the prediction sample set of the K nearest neighbor model.

[0068] Specifically, the K-nearest neighbor model is a machine learning model that uses the KNN algorithm. The idea of ​​the K-Nearest Neighbor (KNN) classification algorithm is to use multiple samples and their corresponding categories to build a prediction sample set; for a new sample to be predicted, in the feature space, if most of the k nearest (i.e., the closest) samples near the new sample belong to a certain category, then the sample also belongs to this category.

[0069] For the K nearest neighbor model used to predict the blood relationship of data, the samples of its prediction sample set include multiple task data, and the category of the sample is the blood relationship corresponding to the task data. Therefore, the first data set can be used as the initial sample set of the K nearest neighbor model. For the instance data in the second data set, it can be first classified based on the initial K nearest neighbor model and the initial sample set to obtain the blood relationship corresponding to the instance data. It is understandable that the blood relationship of the instance data obtained by the untrained initial K nearest neighbor model may have a very low accuracy. As the parameters are adjusted during training, the blood relationship of the instance data can be gradually accurately predicted.

[0070] After that, the instance data and its corresponding blood relationship are merged into the initial sample set to obtain the prediction sample set of the K nearest neighbor model. At this time, the prediction sample set and model parameters of the K nearest neighbor model have been trained and adjusted, and can accurately predict the category of new samples, that is, the blood relationship of new task data.

[0071] In one embodiment, adjusting the parameters of the initial blood relationship extraction model includes:

[0072] Adjust the prediction category of the K nearest neighbor model, adjust the distance measurement method of the K nearest neighbor model, and adjust the K value of the K nearest neighbor model.

[0073] The prediction category is the data lineage relationship corresponding to the task data. In actual work, since work tasks may be added, modified, deleted, etc., the corresponding task data and data lineage relationship will also be adjusted accordingly.

[0074] The distance measurement method is used to calculate the distance between the new sample to be predicted and the samples in the prediction sample set. Common distance measurement methods include: Minkowski distance, Euclidean distance, Manhattan distance, Chebyshev distance and cosine distance. For different types of samples, choosing the appropriate distance measurement method can more accurately characterize the similarity between samples and thus improve the accuracy of classification.

[0075] The K value is a hyperparameter of the K nearest neighbor model. The determination of the k value has a crucial impact on the prediction results of the KNN algorithm. If the k value is relatively small, it is equivalent to predicting the instance with training samples in a smaller field. At this time, the approximation error of the algorithm will be relatively small, because only training samples close to the input instance will have an effect on the prediction result. The disadvantage of a relatively small k value is that the estimation error of the algorithm is relatively large, and the prediction result will be very sensitive to the neighboring points. In other words, if the neighboring points are noise points, the prediction will be wrong. Therefore, a small k value can easily lead to overfitting of the KNN algorithm. Similarly, if the k value is selected to be larger, training samples with a longer distance can also affect the instance prediction results. At this time, the model is relatively robust and will not be affected by individual noise points on the final prediction results. The disadvantage of selecting a larger k value is that the neighbor error of the algorithm will be larger, and points with a longer distance (not similar to the predicted instance) will also affect the prediction results, causing a large deviation in the prediction results. At this time, the model is prone to underfitting.

[0076] Therefore, in actual engineering practice, the k value can be selected by cross-validation, and the k value with the highest accuracy on the test set can be determined as the final algorithm hyperparameter k.

[0077] In one embodiment, the attribute information is parsed to obtain the first data lineage result corresponding to the task data, including:

[0078] Step 402: parse the attribute information to generate an abstract syntax tree corresponding to the attribute information; the abstract syntax tree is used to represent the lexical information, grammatical information and semantic information contained in the attribute information.

[0079] Among them, the abstract syntax tree is an abstract representation of the syntax architecture of the source code. It expresses the syntax structure of the programming language in a tree form, and each node on the tree represents a structure in the source code. By parsing the code of the attribute information, an abstract syntax tree corresponding to the attribute information can be generated. For the attribute information of the working data, the abstract syntax tree is used to represent the lexical information, grammatical information, and semantic information contained in the attribute information.

[0080] Step 404 , divide each abstract syntax tree into hierarchical nodes, and construct a first data lineage result corresponding to the task data according to the dependency relationship between multiple abstract syntax trees with dependency relationships.

[0081] Specifically, the abstract syntax tree can be divided into hierarchical nodes to obtain the dependency relationship between the abstract syntax trees, and the lineage relationship corresponding to the task data can be constructed according to the dependency relationship between the abstract syntax trees to obtain the first data lineage result.

[0082] In one embodiment, constructing a first data lineage result corresponding to the task data according to the dependency relationship between a plurality of abstract syntax trees having dependency relationships may include the following steps:

[0083] Step 502, splitting the abstract syntax tree to obtain multiple sub-statements of the abstract syntax tree;

[0084] Step 504, determining the data kinship relationship included in the abstract syntax tree according to the multiple sub-statements;

[0085] Step 506: When the abstract syntax tree is split, the abstract syntax tree is parsed according to the data lineage relationship to obtain lineage data of the structured query language script.

[0086] Exemplarily, the abstract syntax tree can be split by a segmented search method to obtain multiple sub-statements of different types of the abstract syntax tree. The lineage data target table and identifier contained in the abstract syntax tree are determined according to the multiple sub-statements obtained by splitting, and after the splitting reaches the last priority, it is determined that the current level of the abstract syntax tree has been split, and then the abstract syntax tree is parsed according to the lineage data target table and the identifier to obtain the lineage data corresponding to the code script.

[0087] In order to further explain the solution of the embodiment of the present application, a specific example is given below. Figure 2 As shown, Figure 2 The diagram is a schematic diagram of a system application architecture for implementing a data lineage generation method with a verification mechanism provided by the present application in an embodiment, which includes a scheduling platform, a task execution platform, a data processing platform and a data application.

[0088] Among them, the scheduling platform is a system for managing and executing scheduled or planned tasks, with scheduled task triggering, task dependency management, task scheduling and error handling mechanisms. These functions ensure that tasks can be automatically executed at the scheduled time and carried out in order according to the set dependencies. Any task failure can be retried or alarmed. Users can schedule tasks through simple operations such as dragging and dropping, and view task running status and logs in real time. The scheduling platform helps enterprises realize automated and intelligent data processing through powerful task management, scheduling and monitoring functions.

[0089] The task execution platform is responsible for the actual execution of tasks. It provides different execution environments to meet the execution of different types of tasks, such as shell scripts, python scripts, SQL scripts, Java programs, FlinkSQL, SparkSQL, Datax tasks, etc. The difference from the scheduling platform is that the scheduling platform is responsible for starting tasks. The scheduling platform submits execution tasks to the task execution platform. The actual execution of tasks is on the execution platform, and the execution platform reports the status of tasks to the scheduling platform.

[0090] The data processing platform is responsible for all data processing processes, including data collection, data calculation, data analysis, machine learning, data storage, etc. It is mainly divided into the following modules:

[0091] Data storage module: provides data persistence capabilities and a variety of storage solutions to meet the needs of different application scenarios.

[0092] Data collection module: responsible for collecting various types of data from other systems, including relational data, logs, files, etc., and also undertakes part of the data standardization and data cleaning logic. And write the data into the target database.

[0093] Data computing module: provides various computing engines such as SparkSQL, FlinkSQL, Spark ML, etc., responsible for executing various data processing tasks.

[0094] Machine Learning Module: The machine learning module covers a wide range of algorithms and methods, from supervised learning to unsupervised learning to deep learning. The machine learning module is an integral part of data analysis and artificial intelligence, which enables the function of learning from data and making predictions through various algorithms and methods.

[0095] A data application can be a relatively independent web application that focuses on the data lineage data generated by calculation. It converts the processed data into actual business value and realizes the deep utilization of data through various applications and services.

[0096] It is understandable that the above-mentioned scheduling platform, task execution platform, data processing platform and data application can be configured on the same terminal, or configured in a system including a terminal and a server, and realized through the interaction between the terminal and the server. For example, the scheduling platform can be configured in the task scheduling terminal; the task execution platform and the data processing platform can be configured in a data processing server or a data processing terminal connected to the task scheduling terminal, or can be configured on the task scheduling terminal; the data application can be configured in an independent data application terminal, or can be configured on the task scheduling terminal.

[0097] Based on the above application architecture, the data lineage processing process includes:

[0098] (1) Configuration tasks:

[0099] Create and configure workflows / tasks under one or more specified projects based on business logic on the scheduling platform and task execution platform.

[0100] (2) Extract workflow / task definition static data:

[0101] The data collection module extracts the workflow / task definition static data under all projects of the scheduling platform and task execution platform, and parses it into relational data.

[0102] (3) Workflow / task information storage:

[0103] Write the parsed data in step (2) into the database.

[0104] (4) Calculation of blood relationship:

[0105] According to the workflow / task static data collected in the database, several abstract syntax trees with dependencies are constructed through lexical analysis, grammatical analysis, and semantic analysis. Each tree is divided into hierarchical nodes, and then data lineage relationships are constructed based on several abstract syntax trees with dependencies and different depths.

[0106] (5) Blood relationship data storage:

[0107] The blood relationship data obtained in step (4) is written into the database.

[0108] (6) Extracting dynamic data of workflow / task running instances:

[0109] The data collection module is used to extract dynamic data of daily workflow / task running instances in all projects. When the number of collected days is large enough, there will be a large amount of sample data.

[0110] (7) KNN algorithm calibration data lineage:

[0111] The large amount of original data obtained in step (6) and the data kinship relationship generated in step (4) are used as sample data for machine learning to train the KNN algorithm machine learning task. The trained KNN algorithm recalculates the kinship relationship of the data parsed in step (2) to achieve local calibration of data kinship.

[0112] (8) Blood relationship data storage:

[0113] If there are some data for which the blood relationship results obtained through step (4) and step (7) are different, the blood relationship data obtained through step (7) is written into the database, and the blood relationship data obtained through step (4) is updated and replaced.

[0114] (9) Data display application:

[0115] Based on the blood relationship data generated in step (8), the blood relationship is displayed in a graphical manner.

[0116] The embodiment of the present application provides a data lineage generation method with a verification mechanism, which first uses a common technical implementation method in the industry to generate data lineage, and then uses a large amount of dynamic sample data to achieve data lineage calibration through machine learning technology, thereby improving the accuracy of data lineage. Due to various reasons such as the diversity of data sources, the complexity of data processing, the distribution of data storage, the extensiveness of data applications, and the complexity of technical implementation, the generated data lineage cannot be 100% accurate and the accuracy of data lineage also lacks data support. The present invention extracts a large amount of dynamic data after task execution and uses machine learning technology to enhance the reliability and accuracy of data lineage.

[0117] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0118] Based on the same inventive concept, the embodiment of the present application also provides a data lineage generation device with a verification mechanism for implementing the data lineage generation method with a verification mechanism involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the data lineage generation device with a verification mechanism provided below can refer to the limitations of the data lineage generation method with a verification mechanism above, and will not be repeated here.

[0119] In an exemplary embodiment, Figure 3 As shown, a data lineage generation device with a verification mechanism is provided, comprising:

[0120] The information acquisition module 301 is used to acquire the task data to be classified and the attribute information of the task data;

[0121] The parsing module 302 is used to parse the attribute information to obtain the first data lineage result corresponding to the task data, and store the first data lineage result as the lineage relationship of the task data in the database; the database stores a plurality of task data and their corresponding lineage relationships;

[0122] Extraction module 303, used to input task data and attribute information into a blood relationship extraction model to obtain a second data blood relationship result corresponding to the task data;

[0123] The updating module 304 is used to update the blood relationship in the database to the second data blood relationship result when the first data blood relationship result and the second data blood relationship result are inconsistent.

[0124] In one embodiment, the blood relationship extraction model is obtained by the following method:

[0125] Extracting task data and its corresponding blood relationship in the database to form a first data set;

[0126] Acquire the instance data generated in the workflow to form a second data set;

[0127] The first data set and the second data set are used to train an initial blood relationship extraction model to adjust parameters of the initial blood relationship extraction model to obtain a trained blood relationship extraction model.

[0128] In one embodiment, the blood relationship extraction model is a K nearest neighbor model;

[0129] The initial blood relationship extraction model is trained using the first dataset and the second dataset, including:

[0130] Use the first data set as the initial sample set of the K nearest neighbor model, classify the instance data in the second data set, and obtain the blood relationship corresponding to the instance data;

[0131] The instance data and its corresponding blood relationship are merged into the initial sample set to obtain the predicted sample set of the K nearest neighbor model.

[0132] In one embodiment, adjusting the parameters of the initial blood relationship extraction model includes:

[0133] Adjust the prediction category of the K nearest neighbor model, adjust the distance measurement method of the K nearest neighbor model, and adjust the K value of the K nearest neighbor model.

[0134] In one embodiment, the parsing module is further configured to:

[0135] Parsing the attribute information to generate an abstract syntax tree corresponding to the attribute information; the abstract syntax tree is used to represent the lexical information, grammatical information and semantic information contained in the attribute information;

[0136] Each abstract syntax tree is divided into hierarchical nodes, and a first data lineage result corresponding to the task data is constructed according to the dependency relationship between multiple abstract syntax trees with dependencies.

[0137] In one embodiment, the parsing module is further configured to:

[0138] Split the abstract syntax tree to obtain multiple sub-statements of the abstract syntax tree;

[0139] Determine the data lineage relationship contained in the abstract syntax tree based on multiple sub-statements;

[0140] When the abstract syntax tree is split, the abstract syntax tree is parsed according to the data lineage relationship to obtain the lineage data of the structured query language script.

[0141] Each module in the above-mentioned data lineage generation device with verification mechanism can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0142] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 4As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data lineage generation method with a verification mechanism is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0143] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0144] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0145] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0146] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0148] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data lineage generation method with a verification mechanism, characterized in that: The method comprises: Acquire task data to be classified and attribute information of the task data; Parsing the attribute information to obtain a first data lineage result corresponding to the task data, and storing the first data lineage result as the lineage relationship of the task data in a database; the database stores a plurality of task data and their corresponding lineage relationships; Inputting the task data and the attribute information into a blood relationship extraction model to obtain a second data blood relationship result corresponding to the task data; wherein the blood relationship extraction model is a K nearest neighbor model; When the first data lineage result and the second data lineage result are inconsistent, the lineage relationship in the database is updated to the second data lineage result.

2. The method according to claim 1, characterized in that The blood relationship extraction model is obtained by the following method: Extracting task data and corresponding blood relationships in the database to form a first data set; Acquire the instance data generated in the workflow to form a second data set; The first data set and the second data set are used to train an initial blood relationship extraction model to adjust the parameters of the initial blood relationship extraction model to obtain the trained blood relationship extraction model.

3. The method according to claim 2, characterized in that The using the first data set and the second data set to train an initial blood relationship extraction model comprises: Using the first data set as an initial sample set of the K nearest neighbor model, classifying the instance data in the second data set to obtain the blood relationship corresponding to the instance data; The instance data and its corresponding blood relationship are merged into the initial sample set to obtain a prediction sample set of the K nearest neighbor model.

4. The method according to claim 3, characterized in that The adjusting the parameters of the initial blood relationship extraction model includes: Adjust the prediction category of the K nearest neighbor model, adjust the distance measurement method of the K nearest neighbor model, and adjust the K value of the K nearest neighbor model.

5. The method according to any one of claims 1 to 4, characterized in that: The parsing of the attribute information to obtain a first data lineage result corresponding to the task data includes: Parsing the attribute information to generate an abstract syntax tree corresponding to the attribute information; the abstract syntax tree is used to represent lexical information, grammatical information and semantic information contained in the attribute information; Each of the abstract syntax trees is divided into hierarchical nodes, and a first data lineage result corresponding to the task data is constructed according to the dependency relationship between a plurality of the abstract syntax trees having dependency relationships.

6. The method according to claim 5, characterized in that The step of constructing a first data lineage result corresponding to the task data according to the dependency relationship between the plurality of abstract syntax trees having the dependency relationship includes: Splitting the abstract syntax tree to obtain multiple sub-statements of the abstract syntax tree; Determine the data lineage relationship included in the abstract syntax tree according to the multiple sub-statements; When the abstract syntax tree is split, the abstract syntax tree is parsed according to the data lineage relationship to obtain lineage data of the structured query language script.

7. A data lineage generation device with a verification mechanism, characterized in that: The device comprises: An information acquisition module, used to acquire task data to be classified and attribute information of the task data; A parsing module, used to parse the attribute information to obtain a first data lineage result corresponding to the task data, and store the first data lineage result as the lineage relationship of the task data in a database; the database stores a plurality of task data and their corresponding lineage relationships; An extraction module, used for inputting the task data and the attribute information into a blood relationship extraction model to obtain a second data blood relationship result corresponding to the task data; wherein the blood relationship extraction model is a K nearest neighbor model; An updating module is used to update the blood relationship in the database to the second data blood relationship result when the first data blood relationship result and the second data blood relationship result are inconsistent.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images