Data blood relationship analysis method and equipment

Abstract

The invention provides a data consanguinity analysis method and equipment, and is applied to the field of data governance. The method comprises the steps of obtaining a data dictionary corresponding to a target database; wherein the target database comprises a plurality of data tables; foreign key information is extracted from the target database, the foreign key information is used for representing the relation among the multiple data tables, and the foreign key information is obtained; performing data correlation analysis on the target database based on a correlation coefficient algorithm to obtain a correlation analysis result; and based on the data dictionary, the foreign key information and the correlation analysis result, constructing a data blood relationship map. As the data correlation analysis is performed on the target database based on the correlation coefficient algorithm, not only can the correlation between the data fields be accurately quantified, but also the correlation analysis result for revealing the potential data dependency relationship between the fields and the business logic can be obtained, so that the data correlation analysis efficiency is improved. The data blood relationship map constructed based on the data dictionary, the foreign key information and the correlation analysis result can reveal potential data dependency and business logic, and the accuracy is high.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of data governance, and in particular to a data bloodline analysis method and device. BACKGROUND

[0002] In the data governance scenario, data bloodline analysis between data tables in the database is an important capability, which is widely used in e-commerce platforms, medical and health fields, financial industry, social networks, manufacturing industry, logistics industry and public services, etc.

[0003] When performing data bloodline analysis on a database without business scripts, the database is first connected to collect data dictionary, and the dimension table is identified according to the multi-field uniform root and consistent type, the upper and lower level relationship of the dimension table field is identified according to the same affix and different root, and the related information is stored and retrieved by using a large model knowledge base to complete the analysis.

[0004] Since the data bloodline analysis method does not involve correlation analysis of the database, the accuracy is relatively low. SUMMARY

[0005] Embodiments of the present application provide a data bloodline analysis method, which can improve the accuracy of data bloodline analysis when performing data bloodline analysis on a database without business scripts.

[0006] To achieve the above object, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, the embodiments of the present application provide a data bloodline analysis method, comprising: obtaining a data dictionary corresponding to a target database; wherein the target database comprises a plurality of data tables; extracting foreign key information from the target database, wherein the foreign key information is used to represent the relationship between the plurality of data tables; performing data correlation analysis on the target database based on a correlation coefficient algorithm to obtain a correlation analysis result; and constructing a data bloodline graph based on the data dictionary, the foreign key information and the correlation analysis result.

[0008] Based on the present solution, since the correlation coefficient algorithm can convert the fuzzy association between data fields into a calculable and comparable numerical value to accurately quantify the correlation, and through the covariant mode of data, it can discover the unperceived dependent relationship, and in combination with the business scenario, it can convert the data association into an interpretable business logic, therefore, by performing data correlation analysis on the target database based on the correlation coefficient algorithm, not only the correlation between the data fields can be accurately quantified, but also the correlation analysis result revealing the potential data dependent relationship and business logic between the fields can be obtained. Furthermore, the data bloodline graph constructed based on the data dictionary, the foreign key information and the correlation analysis result can reveal the potential data dependent relationship and business logic, and has high accuracy.

[0009] In some embodiments of the present application, the data correlation analysis on the target database based on the correlation coefficient algorithm obtains a correlation analysis result, which can include: determining target analysis data corresponding to one or more target field groups from the target database; the target field group includes a pair of fields to be subjected to correlation analysis; processing the target analysis data based on the correlation coefficient algorithm to obtain the correlation analysis result.

[0010] Based on the present scheme, the target analysis data corresponding to one or more target field groups is processed based on the correlation coefficient algorithm, which not only accurately quantifies the correlation degree between fields, but also deeply mines the business logic and dependency relationship between fields, thereby generating a correlation analysis result that combines mathematical rigor and business interpretation. In the case where one or more target field groups meet user needs, the fit degree of the analysis result to the actual scene of the user can be significantly improved.

[0011] In some embodiments of the present application, the target analysis data corresponding to one or more target field groups is determined from the target database, including: determining one or more target field groups according to requirement information; preliminarily screening the one or more target field groups based on a data dictionary; in the case where the one or more target field groups pass the screening, obtaining observation data corresponding to the one or more target field groups from the target database; and constructing the target analysis data based on the observation data.

[0012] Based on the present scheme, one or more target field groups are determined according to requirement information, which can ensure that the one or more target field groups meet user needs. The target field groups are effectively verified by the data dictionary, and only when the target field groups pass the preliminary screening, the corresponding observation data is obtained from the target database, and the target analysis data is constructed based on the observation data. In this way, the data acquisition efficiency is improved, invalid data loading is avoided, redundant calculation is reduced, and the reliability and execution efficiency of the analysis process are improved.

[0013] In some embodiments of the present application, the target analysis data is processed based on the correlation coefficient algorithm to obtain the correlation analysis result, including: performing correlation analysis on the target analysis data based on the correlation coefficient algorithm to obtain a correlation coefficient corresponding to each target field group; determining the correlation of each target field group based on the correlation coefficient corresponding to each target field group; and determining the correlation analysis result based on the correlation of each target field group.

[0014] Based on the present scheme, the correlation coefficients between field groups are calculated based on the correlation coefficient algorithm, and the correlation strength is determined according to the correlation coefficient size, which can generate a structured analysis result, providing an interpretable mathematical basis for data relationship quantification and business decision-making.

[0015] In some embodiments of the present application, before obtaining the data dictionary corresponding to the target database, the method further comprises: extracting entry information used for constructing the data dictionary from the target database; and constructing the data dictionary based on the entry information.

[0016] According to the present scheme, by obtaining the entry information of the target database and generating the data dictionary according to the entry information, the systematic collection and standardized arrangement of the database metadata are realized, thereby providing a basic support for subsequent data governance and analysis.

[0017] In some embodiments of the present application, the entry information comprises field name, data type and field description information; and the data dictionary is constructed based on the entry information, comprising: modifying the entry information; and constructing the data dictionary based on the modified entry information, in the case that the modified entry information is approved.

[0018] According to the present scheme, by updating the entry of the target database and dynamically reconstructing the data dictionary after verification, the accuracy and comprehensiveness of the source data can be ensured.

[0019] In some embodiments of the present application, the data bloodline graph is constructed based on the data dictionary, the foreign key information and the correlation analysis result, comprising: constructing a plurality of nodes in the graph database based on the data dictionary; and constructing edges between the nodes based on the foreign key information and the correlation analysis result, to obtain the data bloodline graph.

[0020] According to the present scheme, by converting the elements in the target database into nodes of the bloodline graph through the data dictionary, and constructing the directed and weighted edges through the foreign key information and the correlation analysis result, a data bloodline graph with both topological structure and quantitative relationship can be formed, thereby supporting the full-link tracing of business logic.

[0021] In some embodiments of the present application, the method further comprises: obtaining a data sample in the target database; verifying the data bloodline graph based on the data sample; and determining the target data bloodline graph based on the verification result.

[0022] According to the present scheme, by obtaining a representative sample of the target database through stratified sampling, and comparing the logical consistency between the representative sample and the data bloodline graph, the instantiation verification of the data bloodline graph is realized, and then the target data bloodline graph with more accurate structure and value can be determined based on the verification result.

[0023] In some embodiments of the present application, based on the verification result, the target data blood relationship diagram is determined, including: in the case that the data blood relationship diagram passes the verification, the data blood relationship diagram is taken as the target data blood relationship diagram; in the case that the data blood relationship diagram fails the verification, the target database is updated multiple times, the data correlation of each updated target database is analyzed based on the correlation coefficient algorithm to obtain a correlation analysis result corresponding to each update; based on the correlation analysis result corresponding to each update, a target correlation analysis result corresponding to the target database is determined, and based on the target correlation analysis result, the target data blood relationship diagram is constructed.

[0024] Based on the present scheme, the data blood relationship diagram is verified through data samples, and if the verification passes, the target blood relationship diagram is directly output; if the verification fails, the data correlation is reanalyzed through the correlation coefficient algorithm based on the database updated multiple times. Further, based on the correlation analysis result of the reanalysis, a correlation analysis result accurately and reliably reflecting the dependency relationship can be determined, and finally a blood relationship diagram accurately and reliably reflecting the data and business logic of the dependency relationship is generated, thereby realizing the continuous calibration and traceable management of the data blood relationship.

[0025] In a second aspect, the embodiments of the present application provide a computing device, including: a processor and a memory for storing processor-executable instructions; and the processor is configured to execute the data blood relationship analysis method provided in the first aspect.

[0026] In a third aspect, the embodiments of the present application provide a data blood relationship analysis apparatus, including: an acquisition module configured to acquire a data dictionary corresponding to a target database; wherein the target database includes a plurality of data tables; an extraction module configured to extract relationships between the plurality of data tables from the target database to obtain foreign key information; an analysis module configured to analyze the data correlation of the target database based on a correlation coefficient algorithm to obtain a correlation analysis result; and a blood relationship diagram construction module configured to construct a data blood relationship diagram based on the data dictionary, the foreign key information and the correlation analysis result.

[0027] In a fourth aspect, the embodiments of the present application provide a storage medium, which stores a computer program for executing the data blood relationship analysis method provided in the first aspect.

[0028] In a fifth aspect, the embodiments of the present application provide a computer program product, when the instructions in the computer program product are executed by a processor, the data blood relationship analysis method provided in the first aspect is executed. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A flowchart of a data blood relationship analysis method provided by the embodiments of the present application is shown.

[0030] Figure 2Another data blood relationship analysis method provided by the embodiment of the present application is shown in the flowchart.

[0031] Figure 3 Another data blood relationship analysis method provided by the embodiment of the present application is shown in the flowchart.

[0032] Figure 4 Another data blood relationship analysis method provided by the embodiment of the present application is shown in the flowchart.

[0033] Figure 5 Another data blood relationship analysis method provided by the embodiment of the present application is shown in the flowchart.

[0034] Figure 6 Another data blood relationship analysis method provided by the embodiment of the present application is shown in the flowchart.

[0035] Figure 7 Another data blood relationship analysis method provided by the embodiment of the present application is shown in the flowchart.

[0036] Figure 8 Another data blood relationship analysis method provided by the embodiment of the present application is shown in the flowchart.

[0037] Figure 9 Another data blood relationship analysis method provided by the embodiment of the present application is shown in the flowchart.

[0038] Figure 10 The structure of a computing device provided by the embodiment of the present application is shown in the schematic diagram.

[0039] Figure 11 The structure of a data blood relationship analysis device provided by the embodiment of the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In order to clearly describe the technical solutions in the embodiments of the present application, the first, second, etc. descriptions in the embodiments of the present application are only used for indicating and distinguishing the description objects, and do not have the order, and do not represent the special limitation of the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0041] The related technical terms in the embodiments of the present application will be described below:

[0042] A computing device is an electronic device used to perform computing tasks. Computing devices can include personal computers, servers, embedded computers, and supercomputers, etc. In this application, servers are taken as examples for illustrative purposes. The servers in this application can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as large databases and artificial intelligence platforms. When the server is a server cluster composed of multiple physical servers, or a distributed system, the multiple physical servers can form a blockchain, and each physical server is a node on the blockchain. The physical types of servers can include rack servers, rack servers, high-density servers, graphic processing unit (GPU) servers, tower servers, and blade servers, artificial intelligence (AI) servers, etc. The type of server is not limited in the embodiments of this application.

[0043] Data dictionary: a repository that stores all data elements and their definitions in a centralized system. It contains descriptive information about tables and fields in the database, such as field names, data types, business meanings, etc.

[0044] Data lineage analysis: a technology that traces the origin, purpose and usage of data. It can help us understand where the data comes from, how it is used, and how it affects other data.

[0045] Spearman rank correlation coefficient: a non-parametric statistical method for measuring the degree of dependence between two variables. It does not require data to strictly follow a certain distribution, so it can be widely used in various data scenarios.

[0046] Graph database: a non-relational database suitable for storing and managing complex network structure data. In a graph database, data is stored as nodes and edges, with nodes representing entities and edges representing relationships between entities.

[0047] Big data analysis model: a technology for processing large-scale, complex, and rapidly changing data sets. It can help us extract valuable information from massive raw data.

[0048] Database management system (DBMS): A software that creates, manages, and operates a database. It provides many functions such as query processing, transaction management, security control, etc.

[0049] Metadata: Data that describes other data. For example, in a database, metadata may include table names, field names, field types, and other information.

[0050] Foreign key: A database constraint that ensures the referential integrity between tables. A foreign key in one table corresponds to a primary key in another table.

[0051] The embodiment of the present application provides a data bloodline analysis method, which obtains a data dictionary corresponding to a target database, extracts the relationship between a plurality of data tables in the target database, obtains foreign key information, performs data correlation analysis on the target database based on a correlation coefficient algorithm, obtains a correlation analysis result, and finally constructs a data bloodline graph based on the data dictionary, the foreign key information and the correlation analysis result. Since the correlation coefficient algorithm is used to analyze the data correlation of the target database, the correlation between the data fields can be accurately quantified, and the correlation analysis result revealing the potential data dependency relationship and business logic can be obtained. Therefore, the data bloodline graph constructed based on the data dictionary, the foreign key information and the correlation analysis result can reveal the potential data dependency relationship and business logic, and has high accuracy.

[0052] The data bloodline analysis method provided by the embodiment of the present application can be applied to a computing device, and the type and structure of the computing device are not limited by the embodiment of the present application, and different types and structures of the computing device do not constitute a limitation on the technical solutions provided by the embodiment of the present application. As can be known by those skilled in the art, with the evolution of the type and structure of the computing device, the technical solutions provided by the embodiment of the present application are also applicable to similar technical problems.

[0053] Figure 1 A flowchart of a data bloodline analysis method provided by the embodiment of the present application. The data bloodline analysis method can be applied to a computing device, such as Figure 1 As shown in the figure, the data bloodline analysis method can include the following steps S101 to S104.

[0054] Step S101, obtaining a data dictionary corresponding to a target database.

[0055] The target database includes a plurality of data tables.

[0056] Exemplarily, the target database refers to a database used for data bloodline analysis in a target scenario. The target scenario can include at least one of an e-commerce platform, a medical and health platform, a financial platform, a social network platform, a manufacturing platform, and a logistics platform. In some examples, different target scenarios can correspond to different target databases. For example, if the target scenario is an e-commerce platform, the target database is the database of the e-commerce platform. For another example, if the target scenario is a medical and health platform, the target database is the database of the medical and health platform. For another example, if the target scenario is a logistics platform, the target database is the database of the logistics platform. The embodiments of the present application do not limit the type of target scenario and target database. The embodiments of the present application take the e-commerce platform as the target scenario and the database of the e-commerce platform as the target database for example.

[0057] Taking the database of the e-commerce platform as the target database for example. The plurality of data tables in the target database can include a user information table, an order table, and a commodity table, which can be specifically referred to Table 1, Table 2, and Table 3 as follows.

[0058] Table 1

[0059]

[0060] Table 2

[0061]

[0062] Table 3

[0063]

[0064] The data dictionary, which is equivalent to the specification of the data, can be pre-established and stored in the target database in the form of a data dictionary file (for example, data_dictionary.csv). In some examples, the data dictionary can include |table name|field name|data type|business meaning|data source|purpose|usage|.

[0065] Taking the database of the e-commerce platform as the target database for example. The data dictionary can include:

[0066] |users|user_id|INT|user unique identifier|generated when the user registers|identify the user|as a foreign key|

[0067] |users|name|VARCHAR|user name|user input|display user information|query display|

[0068] |orders|order_id|INT|order unique identifier|system generated|identify the order|as a primary key|

[0069] | orders | user_id | INT | Order user ID | users table | Associated user | Foreign key association |

[0070] | products | product_id | INT | Product unique identifier | System generated | Identify product | Primary key |

[0071] | products | price | DECIMAL | Product price | Product entry | Display product price | Query display |

[0072] …

[0073] After the data dictionary is pre-built and stored in the target database, when the data lineage analysis is performed on the target database, the script of the computing device can read the pre-built and stored data dictionary in the target database.

[0074] Step S102, extract foreign key information from the target database.

[0075] The foreign key information is used to represent the relationship between the plurality of data tables.

[0076] Exemplarily, the foreign key is a constraint in the database for establishing the relationship between tables, indicating that the field or field combination in a table references the primary key or unique key of another table. The foreign key information can be a complete description of the reference relationship, that is, the foreign key information is the relationship between the plurality of data tables. In some examples, the foreign key information can exist in the form of structured metadata, and can be determined when the database is established. For example, the foreign key information can exist in the form of a database system table. For another example, the foreign key information can exist in the form of a visual relationship diagram. The present application does not limit the type of existence of the foreign key information.

[0077] It can be understood that, with reference to Table 1, Table 2 and Table 3, the relationship between Table 1 and Table 2 can include that orders.user_id is a foreign key, associated with users.user_id. And can represent which customer the order belongs to, wherein a user can have multiple orders, but each order only belongs to one user. The relationship between Table 2 and Table 3 can include that orders.product_id is a foreign key, associated with products.product_id. And can represent the goods contained in the order. Wherein one order can contain multiple goods, but each order item in the order corresponds to only one product.

[0078] Taking the target database including Table 1, Table 2 and Table 3 as an example. The foreign key information can be a complete description including the relationship between the above-mentioned Table 1 and Table 2, and the relationship between Table 2 and Table 3. Taking the foreign key information existing in the form of a database system table as an example, the foreign key information can include:

[0079] | Constraint Name | Subtable Name | Subfield Name | Parent Table Name | Parent Field Name |

[0080] | --- | --- | --- | --- | --- |

[0081] | fk_order_user | orders | user_id | users | user_id |

[0082] Thus, the script of the computing device can query the database system table to obtain the above foreign key information revealing the relationship between Table 1 and Table 2, and the relationship between Table 2 and Table 3.

[0083] In step S103, data correlation analysis is performed on the target database based on a correlation coefficient algorithm to obtain a correlation analysis result.

[0084] The correlation coefficient algorithm can include one of a Spearman correlation coefficient algorithm, a Pearson correlation coefficient algorithm, a point two column correlation algorithm, and a multi-series correlation algorithm, and can be determined according to an application scenario of data bloodline analysis. Taking an e-commerce platform as an example, the correlation coefficient algorithm can be the Spearman correlation coefficient algorithm. The type of the correlation coefficient algorithm is not limited in the embodiments of the present application, and the embodiments of the present application are exemplarily described taking the Spearman correlation coefficient algorithm as an example.

[0085] The correlation analysis result can be a complete description including a field to be analyzed, a correlation coefficient, and a correlation description corresponding to the correlation coefficient. Taking the fields to be analyzed including field 1 and field 2 as an example. In some examples, the correlation analysis result can include:

[0086] | Field 1 | Field 2 | Correlation Coefficient | Correlation Description |

[0087] Taking the field 1 as the total_amount column in Table 2 and the field 2 as the price column in Table 3 as an example. The correlation analysis result can include:

[0088] | orders.total_amount | products.price | 0.85 | Strong positive correlation |.

[0089] Exemplarily, the script of the computing device can first determine one or more pairs of fields to be analyzed for correlation analysis from the target database, and then process the observation data corresponding to each pair of fields by the Spearman correlation coefficient algorithm to obtain the correlation analysis result.

[0090] In step S104, a data bloodline graph is constructed based on the data dictionary, the foreign key information, and the correlation analysis result.

[0091] For example, the script of the computing device can first construct multiple nodes based on the data dictionary, foreign key information and correlation analysis results, and then construct the edges between the nodes to obtain a data lineage graph.

[0092] In the data lineage analysis method provided in this application embodiment, the correlation coefficient algorithm can transform fuzzy relationships between data fields into calculable and comparable values ​​to accurately quantify correlations. Furthermore, it can discover unnoticed dependencies through data covariation patterns and, combined with business scenarios, transform data relationships into interpretable business logic. Therefore, performing data correlation analysis on a target database based on the correlation coefficient algorithm not only accurately quantifies the correlations between data fields but also yields correlation analysis results that reveal potential data dependencies and business logic between fields. Furthermore, the data lineage diagram constructed based on the data dictionary, foreign key information, and correlation analysis results can reveal potential data dependencies and business logic with high accuracy.

[0093] like Figure 2 As shown above, in the above Figure 1 Based on the embodiment shown, step S103 performs data correlation analysis on the target database based on the correlation coefficient algorithm to obtain the correlation analysis results, which may include the following steps S1031 and S1032.

[0094] Step S1031: Determine the target analysis data corresponding to one or more target field groups from the target database.

[0095] The target field group includes a pair of fields for which correlation analysis is to be performed.

[0096] In some examples, the target field group can be a pair of fields in the same table. For example, the target field group could include `total_amount` (total order amount, `orders.total_amount`) from Table 2 and `quantity` (quantity of goods, `orders.total_quantity`) from Table 2. In other examples, the target field group can be a pair of fields in different tables. For example, the target field group could include `total_amount` (total order amount, `orders.total_amount`) from Table 2 and `price` (product price, `products.price`) from Table 3.

[0097] The target analysis data can be data of a specific format constructed based on observation data corresponding to one or more target field groups. For example, a target field group includes total_amount (order total amount, orders.total_amount) in Table 2 and price (product price, products.price) in Table 3, and the order total amount corresponds to variable X and the product price corresponds to variable Y.

[0098] The target analysis data can include

[0099] | Order ID | X | Y |

[0100] |---|---|---|

[0101] |1|100|50|

[0102] |2|200|100|

[0103] |3|150|75|

[0104] |4|300|150|

[0105] |5|250|125|

[0106] In some examples, the script of the computing device can first determine one or more target groups according to the requirement information input by the user, and then construct target analysis data based on observation data corresponding to the one or more target groups.

[0107] In step S1032, the target analysis data is processed based on a correlation coefficient algorithm to obtain a correlation analysis result.

[0108] For example, the Spearman correlation coefficient algorithm, the script of the computing device can call the Spearman correlation coefficient algorithm to perform data assignment, rank difference calculation, and Spearman coefficient calculation on the target analysis data, and obtain the correlation analysis result.

[0109] In the data blood relationship analysis method provided in the embodiments of the present application, the target analysis data corresponding to one or more target field groups is processed by the correlation coefficient algorithm, which can not only accurately quantify the correlation degree between fields, but also deeply mine the business logic and dependency relationship between fields, thereby generating a correlation analysis result that has both mathematical rigor and business interpretation. In the case where the one or more target field groups meet the user's requirements, the fit degree of the analysis result to the actual scene of the user can be significantly improved.

[0110] As Figure 3 shown in the above Figure 2Based on the embodiment shown, the step S1031 of determining the target analysis data corresponding to the one or more target field groups from the target database can include the following steps S301 to S304.

[0111] The step S301 of determining the one or more target field groups according to the requirement information.

[0112] The requirement information can be an analysis requirement related to the target field groups proposed by the user in an explicit or implicit manner. In some examples, the requirement information can include the target field groups explicitly specified by the user as required to be analyzed.

[0113] Taking an example in which the requirement information can include the target field groups explicitly specified by the user as required to be analyzed, the script of the computing device can identify the requirement information to obtain the one or more target field groups.

[0114] The step S302 of performing preliminary screening on the one or more target field groups based on the data dictionary.

[0115] The script of the computing device can first explicitly determine the one or more target field groups, and then directly match the same field names in the data dictionary to obtain a matching result. The matching result can include matching to the same field names and not matching to the same field names. The matching to the same field names can indicate that the preliminary screening of the one or more target field groups is passed, and the not matching to the same field names can indicate that the preliminary screening of the one or more target field groups fails or is not passed.

[0116] The step S303 of obtaining the observation data corresponding to the one or more target field groups from the target database in the case where the one or more target field groups pass the screening.

[0117] In the case where the matching to the same field names indicates that the one or more target field groups pass the screening, the script of the computing device can derive the observation data corresponding to the one or more target field groups in the target database.

[0118] The step S304 of constructing the target analysis data based on the observation data.

[0119] Taking an example in which a target field group includes an order total amount (variable X) and a commodity price (variable Y), and the order total amount includes [100 200 150 300 250] and the commodity price includes [50 100 75 150 125], the target analysis data can include

[0120] | Order ID | X | Y |

[0121] | 1 | 100 | 50 |

[0122] | 2 | 200 | 100 |

[0123] |3|150|75|

[0124] |4|300|150|

[0125] |5|250|125|

[0126] The script of the computing device can collate the observation data to obtain target analysis data in a specific format.

[0127] In the data blood relationship analysis method provided in the embodiments of the present application, one or more target field groups are determined according to the requirement information, so that the one or more target field groups can meet the user requirements. The target field groups are verified for effectiveness by using a data dictionary, and the corresponding observation data is obtained from a target database only when the target field groups pass the preliminary screening, and the target analysis data is constructed based on the observation data. In this way, the data acquisition efficiency is improved, invalid data loading is avoided, redundant calculation is reduced, and the reliability and execution efficiency of the analysis process are improved.

[0128] As shown in the above Figure 4 , on the basis of the embodiment shown in the above Figure 2 , step S1032 processes the target analysis data based on a correlation coefficient algorithm to obtain a correlation analysis result, which can include steps S401 to S403.

[0129] In step S401, the target analysis data is analyzed for correlation based on a correlation coefficient algorithm to obtain a correlation coefficient corresponding to each target field group.

[0130] The correlation coefficient can be a numerical value for representing the size of the correlation degree, and one target field group can correspond to one correlation coefficient r s .

[0131] Taking the correlation coefficient algorithm as the Spearman correlation coefficient algorithm, and the target analysis data including

[0132] | Order ID | X | Y |

[0133] | 1 | 100 | 50 |

[0134] | 2 | 200 | 100 |

[0135] | 3 | 150 | 75 |

[0136] | 4 | 300 | 150 |

[0137] | 5 | 250 | 125 |

[0138] For example, the script of the computing device can call the Spearman correlation coefficient algorithm, and assign data values (ranks, also known as levels) according to the size of each observation value in the target analysis data, so as to obtain the sorted data:

[0139] | Order ID | X | Y | rank(X) | rank(Y) |

[0140] | 1 | 100 | 50 | 1 | 1 |

[0141] | 3 | 150 | 75 | 2 | 2 |

[0142] | 2 | 200 | 100 | 3 | 3 |

[0143] | 5 | 250 | 125 | 4 | 4 |

[0144] | 4 | 300 | 150 | 5 | 5 |

[0145] Then, for each pair of observations, the rank difference di of the pair of observations is calculated, where di = rank(Xi) - rank(Yi), and then we have:

[0146] | Order ID | X | Y | rank(X) | rank(Y) | di | di 2 |

[0147] | 1 | 100 | 50 | 1 | 1 | 0 | 0 |

[0148] | 3 | 150 | 75 | 2 | 2 | 0 | 0 |

[0149] | 2 | 200 | 100 | 3 | 3 | 0 | 0 |

[0150] | 5 | 250 | 125 | 4 | 4 | 0 | 0 |

[0151] | 4 | 300 | 150 | 5 | 5 | 0 | 0 |

[0152] Thus, the sum of squares of all rank differences di can be calculated: ∑di 2 = 0 + 0 + 0 + 0 + 0 = 0;

[0153] Further, the Spearman rank correlation coefficient rs can be calculated according to the following formula (1).

[0154] rs = 1 - 6∑di 2 / n*(n 2 - 1) (1).

[0155] Where n is the number of observations, and is 5. Therefore, rs = 1.

[0156] In step S402, the correlation of each target field group is determined based on the correlation coefficients corresponding to the target field groups.

[0157] The correlation coefficient rs can have a preset correspondence with the correlation. For example, when rs = 1, the correlation can be a complete positive correlation, that is, an increase in one variable (X) is accompanied by an increase in another variable (Y). When rs = -1, the correlation can be a complete negative correlation, that is, an increase in one variable is accompanied by a decrease in another variable. When rs = 0, the correlation can be no correlation, that is, the change of one variable is irrelevant to the change of another variable.

[0158] Exemplarily, the script of the computing device can interpret the correlation coefficients corresponding to each target field group according to the preset correspondence between the correlation coefficients and the correlations, to obtain the correlations of each target field group.

[0159] In step S403, the correlation analysis result is determined based on the correlations of each target field group.

[0160] Exemplarily, the script of the computing device can combine the correlation coefficients and the correlations of each target field group to obtain the correlation analysis result.

[0161] In the data bloodline analysis method provided in the embodiments of the present application, the correlation coefficients between field groups are calculated by the correlation coefficient algorithm, and the correlation strength is determined according to the correlation coefficient size, so that a structured analysis result can be generated, and an interpretable mathematical basis can be provided for data relationship quantification and business decision-making.

[0162] As shown in FIG. 1, the data bloodline analysis method further includes the following steps S105 and S106 on the basis of the above-mentioned embodiments. Figure 5 Figure 1 Before step S301 of acquiring the data dictionary corresponding to the target database, the data bloodline analysis method further includes the following steps S105 and S106.

[0163] In step S105, entry information for constructing the data dictionary is extracted from the target database.

[0164] Exemplarily, the script of the computing device can transmit a dictionary construction instruction to the database management system, and extract the entry information for constructing the data dictionary from the target database based on the communication connection between the data management system and the target database.

[0165] In some examples, the dictionary construction instruction refers to a series of database commands, application programming interface (API) calls or configuration files for creating, maintaining and data dictionary or metadata.

[0166] ​The computing device's script can first send database connection information to the database management system to establish a communication connection between the database management system and the target database. After establishing this connection, it then transmits dictionary construction instructions to the database management system based on this connection. The database connection information may include the target database type, hostname, port, username, and password.

[0167] The entry information includes field name, data type, and field description. The field description may include the field's business meaning, constraints, etc.

[0168] The database management system can respond to dictionary construction commands, query the table structure of the target database based on the communication connection between the database management system and the target database, automatically obtain field names and data types, and extract field description information from the target database.

[0169] Step S106: Construct a data dictionary based on the entry information.

[0170] For example, the script of the computing device can first determine the storage method and table structure of the data dictionary, then clean and transform the entry information, and then establish the relationship to obtain the data dictionary.

[0171] In the data lineage analysis method provided in this application embodiment, by obtaining the entry information of the target database and generating a data dictionary based on the entry information, the systematic collection and standardized organization of database metadata is realized, providing basic support for subsequent data governance and analysis.

[0172] like Figure 6 As shown above, in the above Figure 5 Based on the illustrated embodiment, step S106 constructs a data dictionary based on the entry information, which may include the following steps S601 and S602.

[0173] Step S601: Modify the entry information.

[0174] For example, the script of the computing device can transmit entry modification instructions to the database management system, and modify the entry information based on the communication connection between the data management system and the target database.

[0175] Database management systems can provide a graphical user interface (GUI) and allow users to manually add or modify entries in the data dictionary, including data source, purpose, and usage method.

[0176] The script of the computing device can input corresponding entry modification instructions to the database management system in response to the user manually adding or modifying entries in the data dictionary in the GUI. The database management system can add or modify the entries of the data dictionary in the target database in response to the entry modification instructions based on the communication connection between the data management system and the target database.

[0177] Step S602, in the case where the modified entry information passes the review, constructing the data dictionary based on the modified entry information.

[0178] After modifying the entries of the data dictionary, the modified data dictionary can also be reviewed to ensure the accuracy and consistency of the data dictionary information. The implementation of constructing the data dictionary based on the modified entry information can refer to step S106 of the embodiment shown in Figure 5 The embodiment of the present application will not be repeated here.

[0179] In the data bloodline analysis method provided by the embodiment of the present application, the data dictionary is dynamically reconstructed after the target database entry update and the review verification, which can ensure the accuracy and comprehensiveness of the source data.

[0180] In some embodiments of the present application, the script of the computing device can use the big data analysis module to integrate the data dictionary, the foreign key information and the correlation analysis result to construct the data bloodline graph.

[0181] As shown in Figure 7 Based on the above Figure 1 embodiment, step S104 of constructing the data bloodline graph based on the data dictionary, the foreign key information and the correlation analysis result can include the following steps S701 and S702.

[0182] Step S701, constructing a plurality of nodes in the graph database based on the data dictionary.

[0183] Exemplarily, the script of the computing device can use the big data analysis model to define one or more data tables and fields as nodes in the graph database (such as Neo4j). In some examples, the plurality of nodes can include the users table node, the orders table node and the products table node. In other examples, the plurality of nodes can also include the user_id field node of the orders table, the user_id field node of the users table, the orders.total_amount field node and the products.price field node.

[0184] Step S702, constructing edges between nodes based on the foreign key information and the correlation analysis result to obtain the data bloodline graph.

[0185] The computing device's scripts can also use big data analytics models to define data relationships as edges in a graph database, resulting in a data lineage graph. In some examples, the edges between nodes may include foreign key relationships between user_id field nodes in the orders table, foreign key relationships between user_id field nodes in the users table, and relevance edges between the orders.total_amount field node and the products.price field node (e.g., with a weight of 0.85).

[0186] In the data lineage analysis method provided in this application embodiment, elements in the target database are converted into nodes of the lineage graph through a data dictionary, and directed weighted edges are constructed through foreign key information and correlation analysis results, which can form a data lineage graph that combines topological structure and quantitative relationship, supporting full-link traceability of business logic.

[0187] like Figure 8 As shown above, in the above Figure 1 Based on the embodiment shown, after constructing the data lineage graph based on the data dictionary, foreign key information and correlation analysis results in step S104, the data lineage analysis method further includes the following steps S108 to S110.

[0188] Step S108: Obtain data samples from the target database.

[0189] For example, the data sample may be a portion of the data in the target database. In some examples, the data sample may include any number of actual observation data points preset in the target database. In other examples, the data sample may include actual observation data corresponding to one or more target field groups. For example, the sample data may include multiple observation values ​​in orders.total_amount in Table 2 and multiple observation values ​​in products.price in Table 3. This application does not limit the type of data sample; this application provides an example of a data sample including actual observation data corresponding to one or more target field groups.

[0190] Taking a data sample that includes a preset number of actual observation data from the target database as an example, the script of the computing device can randomly extract a preset number of data samples from the target database. Taking a data sample that includes actual observation data corresponding to one or more target field groups as an example, the script of the computing device can first determine the target field groups, and then export a preset number of data samples from the target database that include the actual observation data corresponding to the target field groups.

[0191] Step S109: Verify the data lineage map based on the data samples.

[0192] The script of the computing device can first determine the actual blood relationship of the data sample, compare the actual blood relationship with the data blood relationship graph, obtain a verification result, and thereby verify the data blood relationship graph. The verification result can be in the form of a verification report file, and can include data blood relationship graph verification pass and data blood relationship graph verification failure.

[0193] Taking an example in which the target field group includes two field groups, and are orders.user_id field and users.user_id field, and orders.total_amount field and products.price field respectively, and the data sample includes actual observation data corresponding to the orders.user_id field, the users.user_id field, the orders.total_amount field and the products.price field respectively. The script of the computing device can first determine the actual blood relationship of the orders.user_id field and the users.user_id field, and the actual blood relationship of the orders.total_amount field and the products.price field based on the data sample, and then compare the actual blood relationship with the data blood relationship graph to obtain a verification result. If the data blood relationship graph verification passes, the verification result can include:

[0194] The accuracy of the data blood relationship is 95%

[0195] The discovered error is:

[0196] The foreign key relationship between the user_id field of the orders table and the user_id field of the users table is verified

[0197] The correlation between the orders.total_amount field and the products.price field is verified

[0198] The suggestion is:

[0199] There is no need to adjust the data dictionary and the correlation analysis algorithm.

[0200] If the data blood relationship graph verification fails, the verification result can include:

[0201] The accuracy of the data blood relationship is 45%

[0202] The discovered error is:

[0203] The foreign key relationship between the user_id field of the orders table and the user_id field of the users table is verified

[0204] The correlation validation between the orders.total_amount field and the products.price field failed.

[0205] suggestion:

[0206] Update the database, re-perform correlation analysis, and reconstruct the data lineage graph.

[0207] Step S110: Based on the verification results, determine the target data lineage map.

[0208] For example, the target data lineage map can be a data lineage map with an accuracy reaching a preset percentage. The preset percentage can be any value greater than or equal to 90% and less than 1. For example, the preset percentage can be 95%. This application embodiment does not limit the size of the preset percentage, and this application embodiment uses a preset percentage of 95% as an example for illustrative purposes.

[0209] The computing device's script can determine whether to update the target database and rebuild the data lineage map based on whether the data lineage map verification passes or fails, thus obtaining a target data lineage map that achieves the target accuracy.

[0210] In the data lineage analysis method provided in this application embodiment, a representative sample of the target database is obtained by stratified sampling, and the logical consistency between the representative sample and the data lineage map is compared to realize the instantiation verification of the data lineage map. Based on the verification result, a target data lineage map with more accurate structure and values ​​can be determined.

[0211] like Figure 9 As shown above, in the above Figure 8 Based on the embodiment shown, step S110 determines the target data lineage map based on the verification result, which may include the following steps S901 and S902.

[0212] Step S901: If the data lineage map verification passes, use the data lineage map as the target data lineage map.

[0213] Since the accuracy of data lineage relationships can be greater than the preset percentage when the data lineage diagram passes verification, the data lineage diagram can be directly used as the target data lineage diagram.

[0214] Step S902: If the data lineage map verification fails, update the target database multiple times. Perform data correlation analysis on the target database after each update based on the correlation coefficient algorithm to obtain the correlation analysis results corresponding to each update. Based on the correlation analysis results corresponding to each update, determine the target correlation analysis results corresponding to the target database. Based on the target correlation analysis results, construct the target data lineage map.

[0215] The number of times of updating the target database can be any positive integer greater than or equal to 2. In some examples, the number of times of updating the target database can be 3. In other examples, the number of times of updating the target database can be 5. Embodiments of the present application do not limit the number of times of updating the target database, and embodiments of the present application are exemplarily described by taking the number of times of updating the target database as 5.

[0216] Taking the number of times of updating the target database as 5, the target database updated 5 times by the script of the computing device can obtain the target database updated 5 times.

[0217] The updating of the target database can update part of the data in the target database, or can update all the data in the target database. In some examples, the part of the data in the target database can include the actual observation data corresponding to the target field group whose correlation verification fails. Embodiments of the present application do not limit the update range of the target database, and embodiments of the present application are exemplarily described by taking the part of the data in the target database as an example.

[0218] Since the correlation coefficient algorithm is used to analyze the data correlation of the target database updated each time, the implementation manner of obtaining the correlation analysis result corresponding to each update is similar to the implementation manner of obtaining the correlation analysis result by using the correlation coefficient algorithm to analyze the data correlation of the target database, and details can be referred to Figure 1 The step S103 of the embodiment shown in the figure is not limited by embodiments of the present application.

[0219] Based on the correlation analysis result corresponding to each update, determining the target correlation analysis result corresponding to the target database can include: calculating the average value of the correlation coefficients in the correlation analysis result corresponding to each update, taking the average value as the target correlation coefficient, and determining the target correlation analysis result based on the target correlation coefficient and the correlation corresponding to the target correlation coefficient.

[0220] The implementation manner of determining the target correlation analysis result based on the target correlation coefficient and the correlation corresponding to the target correlation coefficient can be referred to, for example, the step S402 and the step S403 of the embodiment shown in the figure. Figure 4 The step S402 and the step S403 of the embodiment shown in the figure are not repeated by embodiments of the present application.

[0221] Based on the target correlation analysis result, the implementation manner of constructing the target data blood relationship graph is similar to the implementation manner of constructing the data blood relationship graph based on the correlation analysis result, and details can be referred to, for example, the step S104 of the embodiment shown in the figure. Figure 1 The step S104 of the embodiment shown in the figure is not repeated by embodiments of the present application.

[0222] In the data bloodline analysis method provided in the embodiments of the present application, the data sample is used to verify the data bloodline graph, and if the verification is passed, the target bloodline graph is directly output; if the verification fails, the data correlation is reanalyzed based on the correlation coefficient algorithm and the database updated for multiple times. Then, the correlation analysis result that accurately and reliably reflects the dependency relationship can be determined based on the reanalyzed correlation analysis result, and the bloodline graph that accurately and reliably reflects the data and the business logic of the dependency relationship is finally generated, thereby realizing the continuous calibration and traceable management of the data bloodline.

[0223] Corresponding to the foregoing embodiments of the data bloodline analysis method, the present application further provides an embodiment of a computing device. As shown in Figure 10 The hardware entities of the computing device 100 include a processor 1001 and a memory 1002 for storing executable instructions of the processor 1001. The processor 1001 is configured to execute any of the foregoing data bloodline analysis methods.

[0224] With reference to Figure 10 The computing device 1001 can further include a communication interface 1003, wherein,

[0225] The processor 1001 can control the overall operation of the computing device 100.

[0226] The communication interface 1003 can enable the computing device 100 to communicate with other electronic devices or servers through a network.

[0227] The memory 1002 is configured to cache the data to be processed or the data that has been processed by the processor 1001 and the modules in the computing device 100, and can be implemented by FLASH (Flash) or RAM (Random Access Memory, Random Access Memory).

[0228] Corresponding to the foregoing embodiments of the data bloodline analysis method, the present application further provides a data bloodline analysis apparatus, as shown in Figure 11 The data bloodline analysis apparatus 11 can include an acquisition module 111, an extraction module 112, an analysis module 113, and a bloodline graph construction module 114.

[0229] The acquisition module 111 is configured to acquire a data dictionary corresponding to a target database; wherein the target database includes a plurality of data tables.

[0230] The extraction module 112 is configured to extract foreign key information from the target database; wherein the foreign key information is used to represent the relationship between the plurality of data tables.

[0231] The analysis module 113 is configured to perform data correlation analysis on the target database based on a correlation coefficient algorithm to obtain a correlation analysis result.

[0232] The bloodline graph construction module 114 is configured to construct a data bloodline graph based on the data dictionary, the foreign key information, and the correlation analysis result.

[0233] In some embodiments of the present application, the analysis module 112 is specifically configured to determine target analysis data corresponding to one or more target field groups from the target database, the target field group including a pair of fields to be subjected to correlation analysis; and process the target analysis data based on a correlation coefficient algorithm to obtain a correlation analysis result.

[0234] In some embodiments of the present application, the analysis module 112 is specifically configured to determine one or more target field groups according to the requirement information; preliminarily screen the one or more target field groups based on the data dictionary; acquire observation data corresponding to the one or more target field groups from the target database in a case where the one or more target field groups pass the screening; and construct target analysis data based on the observation data.

[0235] In some embodiments of the present application, the analysis module 112 is specifically configured to perform correlation analysis on the target analysis data based on a correlation coefficient algorithm to obtain a correlation coefficient corresponding to each target field group; determine the correlation of each target field group based on the correlation coefficient corresponding to each target field group; and determine a correlation analysis result based on the correlation of each target field group.

[0236] In some embodiments of the present application, the data bloodline analysis apparatus 11 can further include a dictionary construction module configured to extract entry information used for constructing a data dictionary from the target database; and construct the data dictionary based on the entry information.

[0237] In some embodiments of the present application, the entry information includes field name, data type, and field description information, and the dictionary construction module is specifically configured to modify the entry information; and construct the data dictionary based on the modified entry information in a case where the modified entry information passes an audit.

[0238] In some embodiments of the present application, the bloodline graph construction module 114 is specifically configured to construct a plurality of nodes in a graph database based on the data dictionary; and construct edges between the nodes based on the foreign key information and the correlation analysis result to obtain the data bloodline graph.

[0239] In some embodiments of the present application, the data bloodline analysis apparatus 11 can further include a verification module configured to acquire a data sample in the target database; verify the data bloodline graph based on the data sample; and determine a target data bloodline graph based on a verification result.

[0240] In some embodiments of the present application, the checking module is specifically configured to, in a case where the data bloodline graph passes the checking, take the data bloodline graph as a target data bloodline graph; in a case where the data bloodline graph fails the checking, update the target database multiple times, perform data correlation analysis on each time of the updated target database based on a correlation coefficient algorithm to obtain a correlation analysis result corresponding to each time of the update; determine a target correlation analysis result corresponding to the target database based on the correlation analysis result corresponding to each time of the update, and construct the target data bloodline graph based on the target correlation analysis result.

[0241] The beneficial technical effects corresponding to the above-mentioned exemplary embodiments of the data bloodline graph analysis device 11 can refer to the corresponding beneficial technical effects of the method embodiments described above, and will not be repeated here.

[0242] In addition to the above-mentioned devices and apparatuses, the embodiments of the present application can also provide a computer program product comprising computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the data bloodline analysis method of various embodiments of the present application described in the above-mentioned method embodiment part.

[0243] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" language or similar programming languages. The program code can be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0244] In addition, the embodiments of the present application can also be a computer readable storage medium having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps of the data bloodline analysis method of various embodiments of the present application described in the above-mentioned method embodiment part.

[0245] The computer readable storage medium can be any combination of one or more computer readable medium. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium includes, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0246] The above describes the basic principles of the present application in combination with specific embodiments, but the advantages, benefits, effects and the like mentioned in the present application are only examples and are not limiting, and it cannot be considered that the above-mentioned advantages, benefits, effects and the like are necessarily possessed by each embodiment of the present application. In addition, the specific details of the above-described embodiments are only for the purpose of example and for the purpose of understanding, and the above-described details do not limit the present application to necessarily adopt the above-described specific details for implementation.

[0247] Those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and equivalents thereof, the present application is also intended to include these modifications and variations.

[0248] Furthermore, the above-described embodiments are only specific embodiments of the present application and are not intended to limit the scope of protection of the present application, and any modifications, equivalent replacements, improvements and the like made on the basis of the technical solutions of the present application shall be included in the scope of protection of the present application.

Claims

1. A data lineage analysis method, characterized in that, include: Obtain the data dictionary corresponding to the target database; wherein, the target database includes multiple data tables; Foreign key information is extracted from the target database; wherein the foreign key information is used to characterize the relationship between the multiple data tables; Data correlation analysis is performed on the target database based on the correlation coefficient algorithm to obtain the correlation analysis results; A data lineage graph is constructed based on the data dictionary, the foreign key information, and the correlation analysis results.

2. The method according to claim 1, characterized in that, The correlation analysis of the target database based on the correlation coefficient algorithm, to obtain the correlation analysis results, includes: Target analysis data corresponding to one or more target field groups are determined from the target database; the target field group includes a pair of fields to be subjected to correlation analysis. The target analysis data is processed based on the correlation coefficient algorithm to obtain the correlation analysis results.

3. The method according to claim 2, characterized in that, The step of determining the target analysis data corresponding to one or more target field groups from the target database includes: Based on the requirements information, identify one or more target field groups; Based on the data dictionary, perform preliminary screening of one or more target field groups; If the screening of one or more target field groups is passed, the observation data corresponding to one or more target field groups are obtained from the target database; Based on the observation data, the target analysis data is constructed.

4. The method according to claim 2, characterized in that, The process of processing the target analysis data based on the correlation coefficient algorithm to obtain the correlation analysis results includes: Based on the correlation coefficient algorithm, correlation analysis is performed on the target analysis data to obtain the correlation coefficients corresponding to each target field group; The correlation of each target field group is determined based on the correlation coefficient corresponding to each target field group. The correlation analysis results are determined based on the correlation of each of the target field groups.

5. The method according to any one of claims 1-4, characterized in that, Before obtaining the data dictionary corresponding to the target database, the method further includes: Extract entry information for constructing the data dictionary from the target database; The data dictionary is constructed based on the entry information.

6. The method according to claim 5, characterized in that, The entry information includes field name, data type, and field description information; the construction of the data dictionary based on the entry information includes: Modify the entry information; If the modified entry information is approved, the data dictionary is constructed based on the modified entry information.

7. The method according to any one of claims 1-4, characterized in that, The construction of a data lineage graph based on the data dictionary, the foreign key information, and the correlation analysis results includes: Multiple nodes are constructed in the graph database based on the data dictionary; Based on the foreign key information and the correlation analysis results, edges are constructed between the nodes to obtain the data lineage graph.

8. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain data samples from the target database; The pedigree chart is verified based on the data sample. Based on the verification results, the pedigree of the target data is determined.

9. The method according to claim 8, characterized in that, The process of determining the target data lineage based on the verification results includes: If the data lineage graph passes the verification, the data lineage graph will be used as the target data lineage graph. If the data lineage graph verification fails, the target database is updated multiple times. Based on the correlation coefficient algorithm, a data correlation analysis is performed on the target database after each update to obtain the correlation analysis results corresponding to each update. Based on the correlation analysis results corresponding to each update, the target correlation analysis results corresponding to the target database are determined. Based on the target correlation analysis results, the target data lineage graph is constructed.

10. A computing device, characterized in that, include: A processor, and a memory for storing instructions executable by the processor; The processor is configured to execute the data lineage analysis method according to any one of claims 1 to 9.

Images