Data blood relationship analysis method and device based on multivariate data integration and medium

Abstract

The invention discloses a data blood relationship analysis method and device based on multivariate data integration and a medium, and relates to the technical field of data relationship processing. The method comprises the steps that for each target system, target metadata corresponding to the target system is acquired; determining a first association relationship corresponding to all the target systems according to the first relationship identifiers of all the target metadata; determining a second association relationship corresponding to all the target systems according to the second relationship identifier of all the target metadata; and determining a data blood relationship corresponding to all the target systems according to the first association relationship and the second association relationship. According to the method, the data blood relationship can be obtained from various metadata, so that the universality of data blood relationship analysis is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data relationship processing, and in particular to a data kinship analysis method, device and medium based on multivariate data integration. Background Art

[0002] Data lineage refers to the source, flow and change process of data in the system. It describes how data is generated from the original data source, undergoes a series of processing and conversion, and finally forms the current state. Data lineage is an important basis for managing and governing data. By establishing a reasonable data lineage, on the one hand, the source and processing logic of downstream data can be traced, and on the other hand, the impact scope of changes in upstream data can be quickly analyzed, so that change impact warnings and supporting transformations can be carried out in a timely manner.

[0003] The data lineage analysis methods in the prior art mainly include: 1. Tracking the source and conversion process of data by building and maintaining a metadata system, including descriptions of data sources, definitions of data tables, mapping relationships of fields, etc. 2. Automatically record the flow path of data during data extraction, conversion, and loading, and generate a lineage diagram. 3. Use specific lineage tools to parse data lineage. The data lineage in the prior art is often only constructed and analyzed for one data system, and there is only one kind of metadata. When data lineage analysis is required on the basis of multiple data systems, the existing data lineage analysis methods are difficult to obtain data lineage from multiple metadata. It can be seen that the data lineage analysis methods in the prior art are not universal enough. It is necessary to propose a new data lineage analysis method to improve the universality of data lineage analysis. Summary of the invention

[0004] The present invention provides a data lineage relationship analysis method, device and medium based on multi-data integration, which can obtain data lineage relationships from multiple metadata, thereby improving the universality of data lineage relationship analysis.

[0005] In order to solve the above technical problems, the first aspect of the present invention discloses a data kinship analysis method based on multivariate data integration, the method comprising:

[0006] For each target system, obtain target metadata corresponding to the target system;

[0007] Determine, according to the first relationship identifiers of all the target metadata, the first association relationships corresponding to all the target systems; wherein the first relationship identifiers are information used to represent the relationship between data determined based on the first data rule;

[0008] Determine the second association relationship corresponding to all the target systems according to the second relationship identifiers of all the target metadata; wherein the second relationship identifier is information used to represent the relationship between data determined based on the second data rule;

[0009] According to the first association relationship and the second association relationship, the data lineage relationship corresponding to all the target systems is determined.

[0010] As an optional implementation manner, in the first aspect of the present invention, the target metadata corresponding to the target system includes an ETL task configuration table corresponding to the target system, and the first relationship identifier is a one-taskid identifier; and determining the first association relationship corresponding to all the target systems according to the first relationship identifiers of all the target metadata includes:

[0011] Determine the first source table and the first target table of the target data in all the target systems according to the one-taskid identifiers of all the target metadata, and obtain the attribute information of each of the first source tables and the attribute information of each of the first target tables;

[0012] According to the attribute information of all the first source tables and the attribute information of all the first target tables, the first association relationships corresponding to all the target systems are determined.

[0013] As an optional implementation manner, in the first aspect of the present invention, the second relationship identifier is an SQL identifier; and determining the second association relationships corresponding to all the target systems according to the second relationship identifiers of all the target metadata includes:

[0014] Determine the second source table and the second target table of the target data in all the target systems according to the SQL identifiers of all the target metadata, and obtain the attribute information of each of the second source tables and the attribute information of each of the second target tables;

[0015] According to the attribute information of all the second source tables and the attribute information of all the second target tables, the second association relationships corresponding to all the target systems are determined.

[0016] As an optional implementation manner, in the first aspect of the present invention, determining the data lineage relationship corresponding to all the target systems according to the first association relationship and the second association relationship includes:

[0017] Generate an AST abstract syntax tree according to the first association relationship and the second association relationship, wherein the AST abstract syntax tree is used to record the relationship between all target data in the target system;

[0018] Determine the depth of all syntax branches in the AST abstract syntax tree, traverse and parse each syntax branch of the AST abstract syntax tree in order of depth from large to small, and obtain a virtual table corresponding to each node on each syntax branch, wherein the virtual table includes one or more of table name information, column information, sub-virtual table and table condition information;

[0019] The data lineage relationship corresponding to all the target systems is determined according to all the virtual tables.

[0020] As an optional implementation manner, in the first aspect of the present invention, determining the second source table and the second target table of the target data in all the target systems according to the SQL identifiers of all the target metadata includes:

[0021] For each of the SQL identifiers, based on the analysis of the SQL statement of the SQL identifier, determining a complexity parameter of the SQL identifier, wherein the complexity parameter is used to measure the complexity of the SQL command corresponding to the SQL identifier;

[0022] For each of the SQL identifiers, if the complexity parameter of the SQL identifier is greater than a preset complexity threshold, a second source table and a second target table of the target metadata corresponding to the SQL identifier are determined using a regular expression according to the SQL identifier, and the target metadata corresponding to the SQL identifier is determined to be executed metadata;

[0023] For each of the SQL identifiers, if the complexity parameter of the SQL identifier is less than or equal to a preset complexity threshold and the target metadata corresponding to the SQL identifier is not the executed metadata, then based on the SQL statement corresponding to the SQL identifier, determine the second source table and the second target table of the target metadata corresponding to the SQL identifier.

[0024] As an optional implementation, in the first aspect of the present invention, generating an AST abstract syntax tree according to the first association relationship and the second association relationship includes:

[0025] Generate an initial syntax tree according to the first association relationship and the second association relationship, wherein the initial syntax tree is used to represent the association relationship between all target data in the target system through the relationship between nodes;

[0026] For each node on the initial syntax tree, all propagation paths of the node on the initial syntax tree are obtained, where the propagation path is a node branch on the initial syntax tree that contains the node and does not include a branch path; based on the attribute of the target data corresponding to the node, it is determined whether there is an abnormal path in all the propagation paths corresponding to the node; if the abnormal path exists in all the propagation paths corresponding to the node, the abnormal node is determined based on the abnormal path and the attribute of the target data corresponding to each node on the abnormal path;

[0027] After deleting the abnormal node and the downstream nodes corresponding to the abnormal node from the initial syntax tree, an AST abstract syntax tree is obtained.

[0028] As an optional embodiment, in the first aspect of the present invention, the method further comprises:

[0029] Determine, according to the target metadata, an influence parameter corresponding to the first relationship identifier and an influence parameter corresponding to the second relationship identifier in the target metadata, wherein the influence parameter is used to measure the influence degree of the corresponding first relationship identifier or second relationship identifier in the target metadata;

[0030] And, determining the data lineage relationship corresponding to all the target systems according to the first association relationship and the second association relationship includes:

[0031] Determine whether the influence parameter corresponding to the first relationship identifier is greater than the influence parameter corresponding to the second relationship identifier; if it is determined that the influence parameter corresponding to the first relationship identifier is greater than the influence parameter corresponding to the second relationship identifier, determine the initial data lineage relationship corresponding to all the target systems according to the first association relationship, and supplement the relationship between the target data not determined in the initial data lineage relationship according to the second association relationship to obtain the data lineage relationship corresponding to all the target systems;

[0032] If it is determined that the influence parameter corresponding to the first relationship identifier is less than or equal to the influence parameter corresponding to the second relationship identifier, the initial data lineage relationship corresponding to all the target systems is determined based on the second association relationship, and the relationship between the target data not determined in the initial data lineage relationship is supplemented based on the first association relationship to obtain the data lineage relationship corresponding to all the target systems.

[0033] The second aspect of the present invention discloses a data kinship analysis device based on multivariate data integration, the device comprising:

[0034] A data acquisition module, used for acquiring target metadata corresponding to each target system;

[0035] A first analysis module, configured to determine, according to first relationship identifiers of all target metadata, first association relationships corresponding to all target systems; wherein the first relationship identifiers are information for indicating a relationship between data determined based on a first data rule;

[0036] A second analysis module, used to determine the second association relationship corresponding to all the target systems according to the second relationship identifiers of all the target metadata; wherein the second relationship identifier is information used to represent the relationship between data determined based on the second data rule;

[0037] A comprehensive analysis module is used to determine the data lineage relationship corresponding to all the target systems based on the first association relationship and the second association relationship.

[0038] As an optional implementation manner, in the second aspect of the present invention, the target metadata corresponding to the target system includes an ETL task configuration table corresponding to the target system, and the first relationship identifier is a one-taskid identifier; the first analysis module determines the specific manner of the first association relationships corresponding to all the target systems according to the first relationship identifiers of all the target metadata, including:

[0039] Determine the first source table and the first target table of the target data in all the target systems according to the one-taskid identifiers of all the target metadata, and obtain the attribute information of each of the first source tables and the attribute information of each of the first target tables;

[0040] According to the attribute information of all the first source tables and the attribute information of all the first target tables, the first association relationships corresponding to all the target systems are determined.

[0041] As an optional implementation manner, in the second aspect of the present invention, the second relationship identifier is an SQL identifier; the second analysis module determines the specific manner of the second association relationships corresponding to all the target systems according to the second relationship identifiers of all the target metadata, including:

[0042] Determine the second source table and the second target table of the target data in all the target systems according to the SQL identifiers of all the target metadata, and obtain the attribute information of each of the second source tables and the attribute information of each of the second target tables;

[0043] According to the attribute information of all the second source tables and the attribute information of all the second target tables, the second association relationships corresponding to all the target systems are determined.

[0044] As an optional implementation, in the second aspect of the present invention, the comprehensive analysis module determines the specific manner of the data lineage relationship corresponding to all the target systems according to the first association relationship and the second association relationship, including:

[0045] Generate an AST abstract syntax tree according to the first association relationship and the second association relationship, wherein the AST abstract syntax tree is used to record the relationship between all target data in the target system;

[0046] Determine the depth of all syntax branches in the AST abstract syntax tree, traverse and parse each syntax branch of the AST abstract syntax tree in order of depth from large to small, and obtain a virtual table corresponding to each node on each syntax branch, wherein the virtual table includes one or more of table name information, column information, sub-virtual table and table condition information;

[0047] The data lineage relationship corresponding to all the target systems is determined according to all the virtual tables.

[0048] As an optional implementation, in the second aspect of the present invention, the second analysis module determines the specific manner of the second source table and the second target table of all the target data in the target system according to the SQL identifiers of all the target metadata, including:

[0049] For each of the SQL identifiers, based on the analysis of the SQL statement of the SQL identifier, determining a complexity parameter of the SQL identifier, wherein the complexity parameter is used to measure the complexity of the SQL command corresponding to the SQL identifier;

[0050] For each of the SQL identifiers, if the complexity parameter of the SQL identifier is greater than a preset complexity threshold, a second source table and a second target table of the target metadata corresponding to the SQL identifier are determined using a regular expression according to the SQL identifier, and the target metadata corresponding to the SQL identifier is determined to be executed metadata;

[0051] For each of the SQL identifiers, if the complexity parameter of the SQL identifier is less than or equal to a preset complexity threshold and the target metadata corresponding to the SQL identifier is not the executed metadata, then based on the SQL statement corresponding to the SQL identifier, determine the second source table and the second target table of the target metadata corresponding to the SQL identifier.

[0052] As an optional implementation, in the second aspect of the present invention, the specific manner in which the comprehensive analysis module generates the AST abstract syntax tree according to the first association relationship and the second association relationship includes:

[0053] Generate an initial syntax tree according to the first association relationship and the second association relationship, wherein the initial syntax tree is used to represent the association relationship between all target data in the target system through the relationship between nodes;

[0054] For each node on the initial syntax tree, all propagation paths of the node on the initial syntax tree are obtained, where the propagation path is a node branch on the initial syntax tree that contains the node and does not include a branch path; based on the attribute of the target data corresponding to the node, it is determined whether there is an abnormal path in all the propagation paths corresponding to the node; if the abnormal path exists in all the propagation paths corresponding to the node, the abnormal node is determined based on the abnormal path and the attribute of the target data corresponding to each node on the abnormal path;

[0055] After deleting the abnormal node and the downstream nodes corresponding to the abnormal node from the initial syntax tree, an AST abstract syntax tree is obtained.

[0056] As an optional implementation, in the second aspect of the present invention, the device further includes:

[0057] An impact analysis module, configured to determine, according to the target metadata, an impact parameter corresponding to a first relationship identifier and an impact parameter corresponding to a second relationship identifier in the target metadata, wherein the impact parameter is used to measure the impact degree of the corresponding first relationship identifier or second relationship identifier in the target metadata;

[0058] And, the comprehensive analysis module determines the specific manner of the data lineage relationship corresponding to all the target systems according to the first association relationship and the second association relationship, including:

[0059] Determine whether the influence parameter corresponding to the first relationship identifier is greater than the influence parameter corresponding to the second relationship identifier; if it is determined that the influence parameter corresponding to the first relationship identifier is greater than the influence parameter corresponding to the second relationship identifier, determine the initial data lineage relationship corresponding to all the target systems according to the first association relationship, and supplement the relationship between the target data not determined in the initial data lineage relationship according to the second association relationship to obtain the data lineage relationship corresponding to all the target systems;

[0060] If it is determined that the influence parameter corresponding to the first relationship identifier is less than or equal to the influence parameter corresponding to the second relationship identifier, the initial data lineage relationship corresponding to all the target systems is determined based on the second association relationship, and the relationship between the target data not determined in the initial data lineage relationship is supplemented based on the first association relationship to obtain the data lineage relationship corresponding to all the target systems.

[0061] The third aspect of the present invention discloses another data kinship analysis device based on multivariate data integration, the device comprising:

[0062] A memory storing executable program code;

[0063] a processor coupled to the memory;

[0064] The processor calls the executable program code stored in the memory to execute the data blood relationship analysis method based on multivariate data integration disclosed in the first aspect of the present invention.

[0065] The fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the data kinship analysis method based on multivariate data integration disclosed in the first aspect of the present invention.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] The present invention obtains, for each target system, the target metadata corresponding to the target system; determines the first association relationship corresponding to all target systems according to the first relationship identifier of all target metadata; wherein the first relationship identifier is information used to represent the relationship between data determined based on the first data rule; determines the second association relationship corresponding to all target systems according to the second relationship identifier of all target metadata; wherein the second relationship identifier is information used to represent the relationship between data determined based on the second data rule; determines the data lineage relationship corresponding to all target systems according to the first association relationship and the second association relationship. It can be seen that the present invention can determine the first association relationship based on the first relationship identifier of the target metadata; determine the second association relationship based on the second relationship identifier of the target metadata, and finally determine the data lineage relationship corresponding to all target systems according to the first association relationship and the second association relationship; the method can obtain data lineage relationships from a variety of metadata, thereby improving the universality of data lineage relationship analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 It is a flow chart of a method for analyzing data blood relationship based on multivariate data integration disclosed in an embodiment of the present invention;

[0070] Figure 2It is a structural schematic diagram of a data kinship analysis device based on multivariate data integration disclosed in an embodiment of the present invention;

[0071] Figure 3 It is a structural schematic diagram of another data kinship analysis device based on multivariate data integration disclosed in an embodiment of the present invention;

[0072] Figure 4 It is a structural schematic diagram of another data blood relationship analysis device based on multivariate data integration disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or ends.

[0075] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0076] The present invention discloses a data lineage analysis method, device and medium based on multi-data integration. The main inventive concept is: based on the first relationship identifier of the target metadata, a first association relationship is determined; based on the second relationship identifier of the target metadata, a second association relationship is determined, and finally, according to the first association relationship and the second association relationship, the data lineage relationship corresponding to all target systems is determined; the method can obtain data lineage relationships from multiple metadata, thereby improving the universality of data lineage analysis.

[0077] Embodiment 1

[0078] See also Figure 1 , Figure 1 : is a flow chart of a method for analyzing data kinship based on multivariate data integration disclosed in an embodiment of the present invention. Figure 1 The data lineage relationship analysis method based on multivariate data integration described above can be applied to a data lineage relationship analysis device based on multivariate data integration. The data lineage relationship analysis device based on multivariate data integration can be integrated in a cloud server or a local server, and the embodiments of the present invention are not limited thereto. Figure 1 As shown, the data kinship analysis method based on multivariate data integration may include the following operations:

[0079] Step 101: For each target system, obtain target metadata corresponding to the target system.

[0080] In an embodiment of the present invention, different target systems may be different databases, such as databases corresponding to different departments in the same enterprise. Each database generates corresponding metadata during operation, and metadata is data about data. It provides descriptions, explanations, and background information of the data to help people better understand, manage, and use the data. Metadata is like a "navigation map" of data, enabling people to quickly find the data they need. For example, in a large database, metadata can be used to understand the names, field meanings, data types, and other information of each data table, making it easier to determine which data meets specific query requirements. Among them, metadata has different types or forms of existence based on different metadata processing algorithms.

[0081] In an embodiment of the present invention, optionally, the target metadata corresponding to the target system may include an ETL task configuration table corresponding to the target system, and the ETL task configuration table generally includes multiple fields, such as process ID, source file directory, target table, delimiter, character set, etc. These fields together define the path and processing logic of data flow. For example, the process ID is used to uniquely identify each ETL task; the source file directory and the target table specify the original location and loading destination of the data, respectively. ETL tasks support not only simple data handling, but also complex data conversion operations. For example, Flink SQL or other similar tools can be used to achieve rich data conversion and processing, such as JOIN operations, field calculations, etc. These operations can be implemented by specifying corresponding SQL commands or scripts in the ETL configuration table.

[0082] Step 102: Determine the first association relationship corresponding to all target systems according to the first relationship identifiers of all target metadata.

[0083] In an embodiment of the present invention, the first relationship identifier is information used to represent the relationship between data determined based on the first data rule, and the first relationship identifier can be one of a process ID, a source file directory, a target table, a delimiter, a character set, a one-taskid identifier, and an SQL identifier, and the first relationship identifier is different from the second relationship identifier. By parsing the first relationship identifier, the relationship between the target data corresponding to the first relationship identifier in all target systems, that is, the first association relationship, can be determined.

[0084] In an optional embodiment, the first relationship identifier may be a one-taskid identifier, and step 102 may include:

[0085] According to the one-taskid identifier of all target metadata, determine the first source table and the first target table of the target data in all target systems, and obtain the attribute information of each first source table and the attribute information of each first target table;

[0086] According to the attribute information of all the first source tables and the attribute information of all the first target tables, the first association relationships corresponding to all the target systems are determined.

[0087] In this optional embodiment, the target metadata may be a relation table, which is used to store the relationship between upstream and downstream tables between various systems. Among them, the one-taskid identifier (unique task identifier) ​​is the key information to associate these tables. Through the one-taskid identifier, the first source table and the first target table of the target data can be determined. In this optional embodiment, further, the attribute information of each first source table and the attribute information of each first target table can be obtained, and the attribute information is used to represent the attributes of the corresponding source table or target table, such as row information, column information, data definition, condition information, etc. Based on the attribute information, the relationship between the target data corresponding to the first relationship identifier in all target systems can be determined.

[0088] It can be seen that this optional embodiment is based on the one-taskid identifier of the target metadata, and can determine the first source table and the first target table of the target data in all target systems, and the attribute information of each first source table and the first target table, so as to quickly and efficiently determine the first association relationship corresponding to the target system.

[0089] Step 103: Determine the second association relationships corresponding to all target systems according to the second relationship identifiers of all target metadata.

[0090] In an embodiment of the present invention, the second relationship identifier is information used to represent the relationship between data determined based on the second data rule, and the second relationship identifier can be one of a process ID, a source file directory, a target table, a delimiter, a character set, a one-taskid identifier, and an SQL identifier, and the second relationship identifier is different from the first relationship identifier. By parsing the second relationship identifier, the relationship between the target data corresponding to the second relationship identifier in all target systems, that is, the second association relationship, can be determined.

[0091] In an optional embodiment, the second relationship identifier may be an SQL identifier, and step 103 may include:

[0092] Determine the second source table and the second target table of the target data in all the target systems according to the SQL identifiers of all the target metadata, and obtain the attribute information of each second source table and the attribute information of each second target table;

[0093] According to the attribute information of all the second source tables and the attribute information of all the second target tables, the second association relationships corresponding to all the target systems are determined.

[0094] In this optional embodiment, the SQL identifier is a text file containing a series of SQL statements, which is used to manage the database and perform database operations. SQL (Structured Query Language) is a standard language for communicating with relational databases, which allows users to query, update, delete and insert data, as well as manage database objects. Through the SQL identifier, the second source table and the second target table of the target data can be determined. In this optional embodiment, further, the attribute information of each second source table and the attribute information of each second target table can also be obtained, and the attribute information is used to represent the attributes of the corresponding source table or target table, such as row information, column information, data definition, condition information, etc. Based on the attribute information, the relationship between the target data corresponding to the first relationship identifier in all target systems can be determined.

[0095] It can be seen that this optional embodiment can determine the second source table and the second target table of the target data in all target systems, and the attribute information of each second source table and the second target table based on the SQL identifier of the target metadata, so as to quickly and efficiently determine the first association relationship corresponding to the target system. In addition, the SQL identifier has a wide range of use, which can improve the universality of data analysis in this optional embodiment, thereby improving the universality of data lineage relationship analysis.

[0096] Step 104: Determine the data lineage relationship corresponding to all target systems based on the first association relationship and the second association relationship.

[0097] In an embodiment of the present invention, data lineage refers to the source, flow and change process of data in the system. There are many methods for recording and displaying data lineage, such as a data lineage diagram. Optionally, the data lineage can be mapped to the relationship between the source table, the intermediate table and the target table; optionally, determining the data lineage corresponding to all target systems can be to determine the names of the source table, the intermediate table, and the target table corresponding to the target system and the corresponding system ID and database ID, and to construct the association between the tables: record the correspondence between the source table and the intermediate table, and between the intermediate table and the target table. Further optionally, the method may also include: providing a user interface to display the lineage, and supporting interactive query and screening, and may also export the lineage data as an Excel document for further analysis and verification.

[0098] It can be seen that the data lineage analysis method based on multi-data integration in the embodiment of the present invention can determine the first association relationship based on the first relationship identifier of the target metadata; determine the second association relationship based on the second relationship identifier of the target metadata, and finally determine the data lineage relationship corresponding to all target systems based on the first association relationship and the second association relationship; this method can obtain data lineage relationships from multiple metadata, thereby improving the universality of data lineage analysis.

[0099] In an optional embodiment, determining the data lineage relationship corresponding to all target systems according to the first association relationship and the second association relationship may include:

[0100] Generate an AST abstract syntax tree according to the first association relationship and the second association relationship;

[0101] Determine the depth of all syntax branches in the AST abstract syntax tree, traverse and parse each syntax branch of the AST abstract syntax tree in descending order of depth, and obtain the virtual table corresponding to each node on each syntax branch;

[0102] Determine the data lineage relationship corresponding to all target systems based on all virtual tables.

[0103] In this optional embodiment, specifically, the AST abstract syntax tree is used to record the relationship between target data in all target systems. The AST abstract syntax tree is a tree structure that represents the syntax of the source code. By parsing the AST abstract syntax tree, the data lineage relationship can be saved through a virtual table. The virtual table includes one or more of table name information, column information, sub-virtual table and table condition information. Finally, the data lineage relationship corresponding to all target systems is determined based on all virtual tables.

[0104] In this optional embodiment, when parsing the AST abstract syntax tree, the depth-first principle is adopted. Since the syntax branches with greater depth record more data relationships, the syntax branches with greater depth are traversed first to obtain the virtual tables corresponding to the main target data. Subsequently, when traversing the syntax trees with smaller depths, it is only necessary to perform compensatory refinement operations based on the main virtual tables that have been determined, thereby improving the efficiency of parsing the AST abstract syntax tree.

[0105] It can be seen that this optional embodiment parses the data lineage relationship through the AST abstract syntax tree, and first traverses the syntax branches with higher depth during the parsing process, thereby improving the efficiency of the AST abstract syntax tree parsing, thereby improving the efficiency of the data lineage relationship parsing. In addition, by defining a virtual table to save and display the data lineage relationship, the construction of the data lineage relationship is more vivid and intuitive, which is conducive to the subsequent analysis of the data according to the data lineage relationship.

[0106] In this optional embodiment, further optionally, the process of parsing the AST abstract syntax tree may include:

[0107] The AST abstract syntax tree is traversed and parsed in a depth-first manner. First, node preprocessing is performed, and a stack structure is maintained. The first traversed node is pushed into the stack. After all child nodes are parsed, the current node is finally parsed. In the AST abstract syntax tree parsing, the concept of virtual table is defined. Each virtual table contains table name information, column information, child virtual table, table condition information, etc. The outermost virtual table is the top-level virtual table. The information of each subquery is parsed and stored in a virtual table. The columns in the virtual table store the corresponding generated blood parent column list and conditional blood parent column list.

[0108] Specifically, the generated blood parent column can be a column used to generate data in the select expression; the conditional blood parent column can be a column used in data generation conditions such as where / group by / join on. For example, in the SQL statement "select c1 co1 from t1 where c2>1", t1.c1 is the generated blood parent column of col, and t1.c2 is the conditional blood parent column of co1.

[0109] Then maintain a stack of virtual tables, with the parent table of the current virtual table saved at the top of the stack. Each time you traverse into a subquery, push the current virtual table into the stack, create a new virtual table corresponding to the subquery, and update it to the current virtual table. Each time you exit a subquery, take the parent virtual table out of the stack, add the current virtual table to the parent virtual table as a child table, or perform other related operations, and then update the parent virtual table to the current virtual table.

[0110] By parsing the AST abstract syntax tree using the above method, efficient data lineage relationship construction can be achieved.

[0111] In another optional embodiment, determining the second source table and the second target table of the target data in all target systems according to the SQL identifiers of all target metadata may include:

[0112] For each SQL identifier, determining a complexity parameter of the SQL identifier based on an analysis of an SQL statement of the SQL identifier;

[0113] For each SQL identifier, if the complexity parameter of the SQL identifier is greater than a preset complexity threshold, a second source table and a second target table of target metadata corresponding to the SQL identifier are determined using a regular expression according to the SQL identifier, and the target metadata corresponding to the SQL identifier is determined to be executed metadata;

[0114] For each SQL identifier, if the complexity parameter of the SQL identifier is less than or equal to a preset complexity threshold and the target metadata corresponding to the SQL identifier is not executed metadata, the second source table and the second target table of the target metadata corresponding to the SQL identifier are determined according to the SQL statement corresponding to the SQL identifier.

[0115] In this optional embodiment, the complexity parameter is used to measure the complexity of the SQL command corresponding to the SQL identifier. Optionally, the complexity parameter can be measured according to the number of related data involved in the SQL command, or according to the number of operations of the SQL command. In this optional embodiment, if all SQL identifiers are parsed one by one, the amount of calculation is large, and the data relations contained in many SQL statements are repeated. Parsing each SQL representation will generate a lot of repeated calculations, which is inefficient. For more complex SQL identifiers, this optional embodiment uses regular expressions to determine the second source table and the second target table of the target metadata corresponding to the SQL identifier, which can analyze most of the metadata and obtain most of the data relations; for the complexity parameter less than or equal to the preset complexity threshold and the target metadata corresponding to the SQL identifier is not the executed metadata, the second source table and the second target table of the target metadata corresponding to the SQL identifier are determined according to any SQL parsing method, and there is no need to parse the SQL identifier whose complexity parameter is less than or equal to the preset complexity threshold and the target metadata corresponding to the SQL identifier is the executed metadata, which greatly reduces the amount of calculation.

[0116] It can be seen that this optional embodiment can execute different parsing methods on SQL identifiers according to the complexity of the SQL identifiers, thereby reducing the computational complexity of SQL identifier analysis and further improving the efficiency of data lineage relationship parsing.

[0117] In yet another optional embodiment, generating an AST abstract syntax tree according to the first association relationship and the second association relationship may include:

[0118] Generate an initial syntax tree according to the first association relationship and the second association relationship, where the initial syntax tree is used to represent the association relationship between target data in all target systems through the relationship between nodes;

[0119] For each node on the initial syntax tree, all propagation paths of the node on the initial syntax tree are obtained, where the propagation path is a node branch on the initial syntax tree that contains the node and does not include a branch path; based on the attribute of the target data corresponding to the node, determine whether there is an abnormal path in all the propagation paths corresponding to the node; if there is an abnormal path in all the propagation paths corresponding to the node, determine the abnormal node based on the abnormal path and the attribute of the target data corresponding to each node on the abnormal path;

[0120] After deleting the abnormal node and the downstream nodes corresponding to the abnormal node from the initial syntax tree, an AST abstract syntax tree is obtained.

[0121] In this optional embodiment, since the data kinship is analyzed based on the target metadata of multiple target systems, when there are many data samples and they are scattered in different databases, the probability of data relationship errors is high. In addition, the data relationships in different target systems may not correspond to each other, and even the data in different target systems may be misjudged due to similar naming or attributes. In order to reduce the probability of the above situation, in this optional embodiment, all propagation paths corresponding to each node are analyzed to determine whether there is an abnormal path. Optionally, the method for determining the abnormal path may include: determining whether the propagation path of the node conforms to the attributes of the target data corresponding to the node, and if not, determining that the propagation path is an abnormal path. Further, the abnormal node is determined based on the abnormal path, and the abnormal node includes: a node with an incorrect data relationship record, a node whose data naming leads to a misjudgment of the data relationship, a node whose data flow process lacks data, etc.

[0122] It can be seen that this optional embodiment can first establish an initial syntax tree, and then analyze the initial syntax tree to exclude abnormal nodes and downstream nodes corresponding to the abnormal nodes to obtain an AST abstract syntax tree. This reduces the occurrence of data relationship misjudgment caused by various factors, thereby improving the accuracy of data lineage relationship analysis.

[0123] In yet another optional embodiment, the method may further include:

[0124] Determine, according to the target metadata, an influence parameter corresponding to the first relationship identifier and an influence parameter corresponding to the second relationship identifier in the target metadata;

[0125] And, according to the first association relationship and the second association relationship, determining the data lineage relationship corresponding to all target systems, including:

[0126] Determine whether the influence parameter corresponding to the first relationship identifier is greater than the influence parameter corresponding to the second relationship identifier. If it is determined that the influence parameter corresponding to the first relationship identifier is greater than the influence parameter corresponding to the second relationship identifier, determine the initial data lineage relationship corresponding to all target systems according to the first association relationship, and supplement the relationship between the target data not determined in the initial data lineage relationship according to the second association relationship to obtain the data lineage relationship corresponding to all target systems.

[0127] If it is determined that the influence parameter corresponding to the first relationship identifier is less than or equal to the influence parameter corresponding to the second relationship identifier, the initial data lineage relationship corresponding to all target systems is determined based on the second association relationship, and the relationship between the target data not determined in the initial data lineage relationship is supplemented based on the first association relationship to obtain the data lineage relationship corresponding to all target systems.

[0128] In this optional embodiment, the influence parameter is used to measure the influence of the corresponding first relationship identifier or the second relationship identifier in the target metadata. For example, the respective influence parameters can be measured by the proportion of the first relationship identifier or the second relationship identifier in the target metadata. In some cases, there may be mainstream relationship identifiers in different target systems, that is, the most used relationship identifiers. For example, most of the databases corresponding to different departments in the same enterprise use the first relationship identifier, and the second relationship identifier is only used in special scenarios or departments. Therefore, at this time, the first relationship identifier can be used as the main one, and the data relationship recorded by the second relationship identifier can be used to supplement the first relationship identifier. At the same time, there are a large number of repeated relationships between the first relationship identifier and the second relationship identifier. At the same time, the data kinship corresponding to all the target systems is determined based on the first association relationship and the second association relationship, which brings more repeated calculations and reduces efficiency.

[0129] It can be seen that this optional embodiment determines the primary relationship identifier based on the influence parameters corresponding to the first relationship identifier and the second relationship identifier, establishes the initial data lineage relationship based on the primary relationship identifier, and then uses the non-primary relationship identifier to supplement the initial data lineage relationship, thereby reducing the computational complexity of the data lineage relationship analysis process and improving the efficiency of the data lineage relationship analysis.

[0130] Embodiment 2

[0131] See also Figure 2 , Figure 2 Schematic diagram of a data kinship analysis device based on multivariate data integration disclosed in an embodiment of the present invention. Figure 2As shown, the data kinship analysis device based on multivariate data integration may include:

[0132] The data acquisition module 201 is used to acquire target metadata corresponding to each target system;

[0133] The first analysis module 202 is used to determine the first association relationship corresponding to all target systems according to the first relationship identifier of all target metadata; wherein the first relationship identifier is information used to represent the relationship between data determined based on the first data rule;

[0134] The second analysis module 203 is used to determine the second association relationship corresponding to all target systems according to the second relationship identifiers of all target metadata; wherein the second relationship identifier is information used to represent the relationship between data determined based on the second data rule;

[0135] The comprehensive analysis module 204 is used to determine the data lineage relationship corresponding to all target systems according to the first association relationship and the second association relationship.

[0136] It can be seen that the data lineage relationship analysis device based on multi-data integration in the embodiment of the present invention can determine the first association relationship based on the first relationship identifier of the target metadata; determine the second association relationship based on the second relationship identifier of the target metadata, and finally determine the data lineage relationship corresponding to all target systems based on the first association relationship and the second association relationship; this method can obtain data lineage relationships from multiple metadata, thereby improving the universality of data lineage relationship analysis.

[0137] In an optional embodiment, the target metadata corresponding to the target system includes an ETL task configuration table corresponding to the target system, and the first relationship identifier is a one-taskid identifier; the first analysis module 202 determines the specific manner of the first association relationship corresponding to all target systems according to the first relationship identifiers of all target metadata, which may include:

[0138] According to the one-taskid identifier of all target metadata, determine the first source table and the first target table of the target data in all target systems, and obtain the attribute information of each first source table and the attribute information of each first target table;

[0139] According to the attribute information of all the first source tables and the attribute information of all the first target tables, the first association relationships corresponding to all the target systems are determined.

[0140] It can be seen that this optional embodiment is based on the one-taskid identifier of the target metadata, and can determine the first source table and the first target table of the target data in all target systems, and the attribute information of each first source table and the first target table, so as to quickly and efficiently determine the first association relationship corresponding to the target system.

[0141] In another optional embodiment, the second relationship identifier is an SQL identifier; the second analysis module 203 determines the specific manner of the second association relationship corresponding to all target systems according to the second relationship identifiers of all target metadata, which may include:

[0142] Determine the second source table and the second target table of the target data in all the target systems according to the SQL identifiers of all the target metadata, and obtain the attribute information of each second source table and the attribute information of each second target table;

[0143] According to the attribute information of all the second source tables and the attribute information of all the second target tables, the second association relationships corresponding to all the target systems are determined.

[0144] It can be seen that this optional embodiment can determine the second source table and the second target table of the target data in all target systems, and the attribute information of each second source table and the second target table based on the SQL identifier of the target metadata, so as to quickly and efficiently determine the first association relationship corresponding to the target system. In addition, the SQL identifier has a wide range of use, which can improve the universality of data analysis in this optional embodiment, thereby improving the universality of data lineage relationship analysis.

[0145] In another optional embodiment, the comprehensive analysis module 204 determines the specific manner of the data lineage relationship corresponding to all target systems according to the first association relationship and the second association relationship, which may include:

[0146] Generate an AST abstract syntax tree according to the first association relationship and the second association relationship, where the AST abstract syntax tree is used to record the relationship between target data in all target systems;

[0147] Determine the depth of all syntax branches in the AST abstract syntax tree, traverse and parse each syntax branch of the AST abstract syntax tree in descending order of depth, and obtain a virtual table corresponding to each node on each syntax branch, where the virtual table includes one or more of table name information, column information, sub-virtual table, and table condition information;

[0148] Determine the data lineage relationship corresponding to all target systems based on all virtual tables.

[0149] It can be seen that this optional embodiment parses the data lineage relationship through the AST abstract syntax tree, and first traverses the syntax branches with higher depth during the parsing process, thereby improving the efficiency of the AST abstract syntax tree parsing, thereby improving the efficiency of the data lineage relationship parsing. In addition, by defining a virtual table to save and display the data lineage relationship, the construction of the data lineage relationship is more vivid and intuitive, which is conducive to the subsequent analysis of the data according to the data lineage relationship.

[0150] In yet another optional embodiment, the second analysis module 203 determines the specific manner of the second source table and the second target table of the target data in all target systems according to the SQL identifiers of all target metadata, which may include:

[0151] For each SQL identifier, based on the analysis of the SQL statement of the SQL identifier, a complexity parameter of the SQL identifier is determined, where the complexity parameter is used to measure the complexity of the SQL command corresponding to the SQL identifier;

[0152] For each SQL identifier, if the complexity parameter of the SQL identifier is greater than a preset complexity threshold, a second source table and a second target table of target metadata corresponding to the SQL identifier are determined using a regular expression according to the SQL identifier, and the target metadata corresponding to the SQL identifier is determined to be executed metadata;

[0153] For each SQL identifier, if the complexity parameter of the SQL identifier is less than or equal to a preset complexity threshold and the target metadata corresponding to the SQL identifier is not executed metadata, the second source table and the second target table of the target metadata corresponding to the SQL identifier are determined according to the SQL statement corresponding to the SQL identifier.

[0154] It can be seen that this optional embodiment can execute different parsing methods on SQL identifiers according to the complexity of the SQL identifiers, thereby reducing the computational complexity of SQL identifier analysis and further improving the efficiency of data lineage relationship parsing.

[0155] In another optional embodiment, the specific manner in which the comprehensive analysis module 204 generates the AST abstract syntax tree according to the first association relationship and the second association relationship may include:

[0156] Generate an initial syntax tree according to the first association relationship and the second association relationship, where the initial syntax tree is used to represent the association relationship between target data in all target systems through the relationship between nodes;

[0157] For each node on the initial syntax tree, all propagation paths of the node on the initial syntax tree are obtained, where the propagation path is a node branch on the initial syntax tree that contains the node and does not include a branch path; based on the attribute of the target data corresponding to the node, determine whether there is an abnormal path in all the propagation paths corresponding to the node; if there is an abnormal path in all the propagation paths corresponding to the node, determine the abnormal node based on the abnormal path and the attribute of the target data corresponding to each node on the abnormal path;

[0158] After deleting the abnormal node and the downstream nodes corresponding to the abnormal node from the initial syntax tree, an AST abstract syntax tree is obtained.

[0159] It can be seen that this optional embodiment can first establish an initial syntax tree, and then analyze the initial syntax tree to exclude abnormal nodes and downstream nodes corresponding to the abnormal nodes to obtain an AST abstract syntax tree. This reduces the occurrence of data relationship misjudgment caused by various factors, thereby improving the accuracy of data lineage relationship analysis.

[0160] In yet another optional embodiment, Figure 3 As shown, the device may also include:

[0161] An impact analysis module 205 is used to determine, according to the target metadata, an impact parameter corresponding to the first relationship identifier and an impact parameter corresponding to the second relationship identifier in the target metadata, wherein the impact parameter is used to measure the impact degree of the corresponding first relationship identifier or second relationship identifier in the target metadata;

[0162] Furthermore, the comprehensive analysis module 204 determines the specific manner in which the data blood relationship corresponding to all target systems is determined according to the first association relationship and the second association relationship, which may include:

[0163] Determine whether the influence parameter corresponding to the first relationship identifier is greater than the influence parameter corresponding to the second relationship identifier. If it is determined that the influence parameter corresponding to the first relationship identifier is greater than the influence parameter corresponding to the second relationship identifier, determine the initial data lineage relationship corresponding to all target systems according to the first association relationship, and supplement the relationship between the target data not determined in the initial data lineage relationship according to the second association relationship to obtain the data lineage relationship corresponding to all target systems.

[0164] If it is determined that the influence parameter corresponding to the first relationship identifier is less than or equal to the influence parameter corresponding to the second relationship identifier, the initial data lineage relationship corresponding to all target systems is determined based on the second association relationship, and the relationship between the target data not determined in the initial data lineage relationship is supplemented based on the first association relationship to obtain the data lineage relationship corresponding to all target systems.

[0165] It can be seen that this optional embodiment determines the primary relationship identifier based on the influence parameters corresponding to the first relationship identifier and the second relationship identifier, establishes the initial data lineage relationship based on the primary relationship identifier, and then uses the non-primary relationship identifier to supplement the initial data lineage relationship, thereby reducing the computational complexity of the data lineage relationship analysis process and improving the efficiency of the data lineage relationship analysis.

[0166] Embodiment 3

[0167] See also Figure 4 , Figure 4 FIG. 1 is a schematic diagram of the structure of another data kinship analysis device based on multivariate data integration disclosed in an embodiment of the present invention. Figure 4 As shown, the data kinship analysis device based on multivariate data integration may include:

[0168] A memory 301 storing executable program codes;

[0169] a processor 302 coupled to the memory 301;

[0170] The processor 302 calls the executable program code stored in the memory 301 to execute the steps of the data kinship analysis method based on multivariate data integration described in the first embodiment of the present invention.

[0171] Embodiment 4

[0172] An embodiment of the present invention discloses a computer storage medium storing computer instructions. When the computer instructions are called, they are used to execute the steps of the data kinship analysis method based on multivariate data integration described in the first embodiment of the present invention.

[0173] Embodiment 5

[0174] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps of the data kinship analysis method based on multivariate data integration described in Embodiment 1.

[0175] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0176] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0177] Finally, it should be noted that the data blood relationship analysis method, device and medium based on multi-data integration disclosed in the embodiments of the present invention are only the preferred embodiments of the present invention, which are only used to illustrate the technical scheme of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical schemes described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical schemes from the spirit and scope of the technical schemes of the embodiments of the present invention.

Claims

1. A data kinship analysis method based on multivariate data integration, characterized in that: The method comprises: For each target system, obtain target metadata corresponding to the target system; Determine, according to the first relationship identifiers of all the target metadata, the first association relationships corresponding to all the target systems; wherein the first relationship identifiers are information used to represent the relationship between data determined based on the first data rule; Determine the second association relationship corresponding to all the target systems according to the second relationship identifiers of all the target metadata; wherein the second relationship identifier is information used to represent the relationship between data determined based on the second data rule; According to the first association relationship and the second association relationship, the data lineage relationship corresponding to all the target systems is determined.

2. The method for analyzing data kinship based on multivariate data integration according to claim 1, characterized in that: The target metadata corresponding to the target system includes an ETL task configuration table corresponding to the target system, and the first relationship identifier is a one-taskid identifier; The determining, according to the first relationship identifiers of all the target metadata, the first association relationships corresponding to all the target systems includes: Determine the first source table and the first target table of the target data in all the target systems according to the one-taskid identifiers of all the target metadata, and obtain the attribute information of each of the first source tables and the attribute information of each of the first target tables; According to the attribute information of all the first source tables and the attribute information of all the first target tables, the first association relationships corresponding to all the target systems are determined.

3. The method for analyzing data kinship based on multivariate data integration according to claim 2 is characterized in that: The second relation identifier is a SQL identifier; The determining, according to the second relationship identifiers of all the target metadata, the second association relationships corresponding to all the target systems includes: Determine the second source table and the second target table of the target data in all the target systems according to the SQL identifiers of all the target metadata, and obtain the attribute information of each of the second source tables and the attribute information of each of the second target tables; According to the attribute information of all the second source tables and the attribute information of all the second target tables, the second association relationships corresponding to all the target systems are determined.

4. The method for analyzing data blood relationship based on multivariate data integration according to claim 3 is characterized in that: Determining the data lineage relationship corresponding to all the target systems according to the first association relationship and the second association relationship includes: Generate an AST abstract syntax tree according to the first association relationship and the second association relationship, wherein the AST abstract syntax tree is used to record the relationship between all target data in the target system; Determine the depth of all syntax branches in the AST abstract syntax tree, traverse and parse each syntax branch of the AST abstract syntax tree in order of depth from large to small, and obtain a virtual table corresponding to each node on each syntax branch, wherein the virtual table includes one or more of table name information, column information, sub-virtual table and table condition information; The data lineage relationship corresponding to all the target systems is determined according to all the virtual tables.

5. The method for analyzing data blood relationship based on multivariate data integration according to claim 2, characterized in that: Determining the second source table and the second target table of all target data in the target system according to the SQL identifiers of all the target metadata includes: For each of the SQL identifiers, based on the analysis of the SQL statement of the SQL identifier, determining a complexity parameter of the SQL identifier, wherein the complexity parameter is used to measure the complexity of the SQL command corresponding to the SQL identifier; For each of the SQL identifiers, if the complexity parameter of the SQL identifier is greater than a preset complexity threshold, a second source table and a second target table of the target metadata corresponding to the SQL identifier are determined using a regular expression according to the SQL identifier, and the target metadata corresponding to the SQL identifier is determined to be executed metadata; For each of the SQL identifiers, if the complexity parameter of the SQL identifier is less than or equal to a preset complexity threshold and the target metadata corresponding to the SQL identifier is not the executed metadata, then based on the SQL statement corresponding to the SQL identifier, determine the second source table and the second target table of the target metadata corresponding to the SQL identifier.

6. The method for analyzing data blood relationship based on multivariate data integration according to claim 4 is characterized in that: Generating an AST abstract syntax tree according to the first association relationship and the second association relationship includes: Generate an initial syntax tree according to the first association relationship and the second association relationship, wherein the initial syntax tree is used to represent the association relationship between all target data in the target system through the relationship between nodes; For each node on the initial syntax tree, all propagation paths of the node on the initial syntax tree are obtained, where the propagation path is a node branch on the initial syntax tree that contains the node and does not include a branch path; based on the attribute of the target data corresponding to the node, it is determined whether there is an abnormal path in all the propagation paths corresponding to the node; if the abnormal path exists in all the propagation paths corresponding to the node, the abnormal node is determined based on the abnormal path and the attribute of the target data corresponding to each node on the abnormal path; After deleting the abnormal node and the downstream nodes corresponding to the abnormal node from the initial syntax tree, an AST abstract syntax tree is obtained.

7. The method for analyzing data blood relationship based on multivariate data integration according to claim 4 is characterized in that: The method further comprises: Determine, according to the target metadata, an influence parameter corresponding to the first relationship identifier and an influence parameter corresponding to the second relationship identifier in the target metadata, wherein the influence parameter is used to measure the influence degree of the corresponding first relationship identifier or second relationship identifier in the target metadata; And, determining the data lineage relationship corresponding to all the target systems according to the first association relationship and the second association relationship includes: Determine whether the influence parameter corresponding to the first relationship identifier is greater than the influence parameter corresponding to the second relationship identifier; if it is determined that the influence parameter corresponding to the first relationship identifier is greater than the influence parameter corresponding to the second relationship identifier, determine the initial data lineage relationship corresponding to all the target systems according to the first association relationship, and supplement the relationship between the target data not determined in the initial data lineage relationship according to the second association relationship to obtain the data lineage relationship corresponding to all the target systems; If it is determined that the influence parameter corresponding to the first relationship identifier is less than or equal to the influence parameter corresponding to the second relationship identifier, the initial data lineage relationship corresponding to all the target systems is determined based on the second association relationship, and the relationship between the target data not determined in the initial data lineage relationship is supplemented based on the first association relationship to obtain the data lineage relationship corresponding to all the target systems.

8. A data kinship analysis device based on multivariate data integration, characterized in that: The device comprises: A data acquisition module, used for acquiring target metadata corresponding to each target system; A first analysis module, configured to determine, according to first relationship identifiers of all target metadata, first association relationships corresponding to all target systems; wherein the first relationship identifiers are information for indicating a relationship between data determined based on a first data rule; A second analysis module, used to determine the second association relationship corresponding to all the target systems according to the second relationship identifiers of all the target metadata; wherein the second relationship identifier is information used to represent the relationship between data determined based on the second data rule; A comprehensive analysis module is used to determine the data lineage relationship corresponding to all the target systems based on the first association relationship and the second association relationship.

9. A data kinship analysis device based on multivariate data integration, characterized in that: The device includes: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the data blood relationship analysis method based on multivariate data integration as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, which, when called, are used to execute the data kinship analysis method based on multivariate data integration as described in any one of claims 1-7.

Images