A cross-system metadata bloodline automatic analysis method

Abstract

The application discloses a kind of cross-system metadata blood relation automatic analysis methods, comprising: (1), cross-system blood relation file and configuration file acquisition step;(2), the component identification and relationship construction step of cross-system blood relation file;(3), data information analysis step;(4), the step of building data blood relation;(5), the visual display step of data blood relation, comprising: the wireframe diagram of the data blood relation built is generated and is shown.The cross-system metadata blood relation automatic analysis method of the application realizes the accurate analysis to the key component in the number of works by the automatic reading of all engineering cross-system blood relation file, and completes the construction of end-point-line-end-point structure on the relationship of data flow between key components, solves the automatic adaptation identification problem of flexible combination between different components in the number of works.

Description

Technical Field

[0001] This invention belongs to the field of data lineage analysis, and is a method for automated lineage analysis across systems. Background Technology

[0002] With the advent of the big data era, the popularization of information technology, and the continuous advancement of digital transformation, taking the banking and internet industries as examples, data governance is being implemented to safeguard the data assets and resources of enterprises and banks, and data lineage is of paramount importance in data governance.

[0003] Data lineage refers to the complete path of data from its source to its destination. Tracking data lineage is crucial in enterprise data management. It records how data is processed, ensuring compliance and accuracy throughout the data lifecycle, thereby improving data quality and decision reliability. Furthermore, data lineage helps enterprise managers analyze and optimize data processing processes, improving efficiency and reducing costs. Data lineage traces the path of data from its source to its destination, identifying which systems, applications, and processes use the data along that path. This is essential for data auditing, compliance, and security. Data lineage also helps enterprises understand the origin, purpose, and value of their data. Through lineage analysis, enterprises can understand which data is most important, which no longer needs to be retained, which can be used for new business scenarios or data science projects, and how to quickly identify databases associated with changed data when changes occur, enabling them to make the same changes.

[0004] Within an enterprise, multiple systems typically exist, and the data interaction and integration between these systems are complex, with data potentially being processed and transformed by multiple systems. In this context, tracing data lineage requires encompassing multiple systems and stages, spanning across multiple systems to determine the complete data path. Data formats and representations may also differ between systems, necessitating data mapping and transformation. Building cross-system data lineages is more challenging than building them within a single system because transferring data between different systems requires navigating different technologies and protocols. Different systems may use different data formats, standards, and languages, further increasing the complexity and challenge of cross-system data lineages. However, building cross-system data lineages is crucial because only by understanding the entire data flow can data quality and integrity be ensured, and the efficiency and reliability of data governance improved.

[0005] As a visual data integration tool, DataStage has been widely used for data extraction, transformation, and loading operations between various database systems. In this process, a large number of data reference relationships, or data lineages, are usually generated. From a metadata perspective, this specifically includes hierarchical reference relationships between various metadata objects, such as: system to system, database to database, table to table, field to field, and lineage to lineage source file mutual reference relationships.

[0006] Current research on data lineage analysis mainly focuses on the accurate parsing of SQL content within databases within the same system. Research on lineage analysis across systems using the DataStage data integration tool is scarce. Furthermore, data lineage analysis using DataStage remains reliant on manual analysis by specific technical personnel or on offline manual ledger records. There is no clear, systematic technical approach to automatically parse lineage relationships within the data integration tool. Manual lineage analysis is difficult, time-consuming, and often inaccurate, leading to frequent production change issues and severely impacting banking operations. Meanwhile, banks now generate massive amounts of data extraction operations using DataStage. To reduce the impact of frequent queries to the DataStage server during online real-time lineage analysis on extraction performance, this invention implements automatic parsing of extraction operation files using an offline parsing method. This ensures accuracy and timeliness while minimizing the impact on system efficiency. Simultaneously, based on the parsed lineage relationships, this invention establishes the relationship between lineage relationships and corresponding extraction operation files, solving the problem of analyzing the actual source of lineage relationships. How to automatically and accurately resolve data lineage relationships across systems, build a complete lineage chain across the entire enterprise system, reduce the difficulty for developers in analyzing cross-system data issues, and ensure the efficient implementation of bank data governance has become an urgent problem to be solved in the field of data lineage analysis and application. Summary of the Invention

[0007] This invention addresses the technical problem in existing technologies that cannot automatically find lineage relationships across systems by proposing an automatic cross-system metadata lineage resolution method, which can solve the above-mentioned problem.

[0008] To achieve the above-mentioned objectives, the present invention employs the following technical solution:

[0009] A method for automatic cross-system metadata lineage resolution includes:

[0010] (1) Steps for obtaining cross-system lineage files and configuration files, including:

[0011] Retrieve all projects under the cross-system data integration tool DataStage;

[0012] Obtain all data extraction jobs under the project, and the data extraction job file must record at least the data flow information of the component;

[0013] (2) The steps for component identification and relationship construction of cross-system lineage files include:

[0014] (21) Identify all components in the sampling operation, and obtain the data flow information of the components and the relationship between the components;

[0015] (22) Based on the data flow information of the components, the types of each component are defined as source components, intermediate components and target components, and the three types of components are constructed into an endpoint-line-endpoint data structure. The source component is a component with only data outflow, the target component is a component with only data inflow, and the intermediate component is a component with both data outflow and data inflow. In the constructed endpoint-line-endpoint data structure, the endpoint is a component with data inflow or outflow, and the line records the data flow information and the data content that flows through it.

[0016] (3) Data information parsing steps, including: parsing the specific data transmitted between components, including parsing the metadata information carried by each component and the correspondence between the metadata information, wherein the metadata information includes library information, table information and field information;

[0017] (4) The steps of constructing data lineage include: sorting out the relationship between metadata based on the parsed data information, and generating corresponding data lineage pairs based on the data flow information and the metadata information to complete the construction of data lineage.

[0018] (5) The steps for visualizing data lineage relationships include: generating wireframe diagrams of the constructed data lineage relationships and displaying them.

[0019] In some embodiments, the method for step (21) of identifying all components in the sampling operation and obtaining data flow information of the components and the relationships between the components includes:

[0020] (211) Iterate through each sampling operation and read all endpoints in the sampling operation;

[0021] (212) Identify endpoint information, including identifying the name and type of the components involved in the endpoint and finding the job name;

[0022] (213) Identify the relationship between endpoints and lines. After completing component identification, identify the relationship between components and lines. Specifically define the structure as: job name. component name. line name.

[0023] In some embodiments, the method for constructing the endpoint-line-endpoint data structure in step (22) includes:

[0024] The data flow information of the line is obtained based on the line name. The data flow information includes at least the components connected to both ends of the line, and an endpoint-line-endpoint data structure is established accordingly.

[0025] In some embodiments, the method for parsing the specific data transmitted between components in step (3) includes:

[0026] (31) Locate the component based on its name;

[0027] (32) Locate the data information of the source component, target component, and intermediate component according to the type of the component. The data information of the source component is stored in the connector and XMLProperties tags, the data information of the target component is stored in the target, connector, and XMLProperties tags, and the data processing information of the intermediate component is recorded in the information of its corresponding output line.

[0028] (33) Parse the data information and data processing information of the source component, target component, and intermediate component respectively;

[0029] (34) This completes the parsing of database, table, and field information in the source and target components, the parsing of the corresponding relationships in the intermediate component data processing, and stores this information in the output lines corresponding to the source and intermediate components. In some embodiments, the method for parsing the data information of the source component in step (33) includes:

[0030] (3311) Parse the database information to which the table read by the source component belongs, including: determining the connection method of the database, determining the database type, and parsing and determining the schema information of the database according to the connection method of the database;

[0031] (3312) Parse the table information read by the source component;

[0032] (3313) Parse the field information of the table read by the source component;

[0033] In step (3311), when the database connection method is JDBC connection, the content of the DataSource tag and Username tag is parsed, and when the content under the tag is not a parameter value, the tag value is read directly to parse out the corresponding Schema information;

[0034] When the content under the tag is a parameter, the specific parameter value is queried through the parameter set, and the database schema information is obtained from the specific parameter value. The parameter set can be obtained by identifying the pst file under the isx project file. The file name of the pst file is the name of the parameter set. When the specific value corresponding to the parameter is a marker that references the environment variable, the corresponding environment variable is obtained through the project to parse the specific parameter value.

[0035] When the database connection method is ODBC, the value of the DataSource tag is parsed to obtain the ODBC data source name. The URL information corresponding to the data source and the information of the connected user are obtained through the ODBC configuration file, thereby obtaining the schema information of the database.

[0036] In some embodiments, parsing the table information read by the source component in step (3312) includes:

[0037] The content of the SelectStatement tag under the XMLProperties tag is parsed, which records the specific database, table, and field information queried by the source component. If the SelectStatement tag is empty, it is obtained through the content of the Table tag under the XMLProperties tag. If the Table tag is empty, the table information in the source component is parsed through the SQL parser in the SQL content under the SelectStatement tag.

[0038] When the parsed Schema name is inconsistent with the Schema name parsed in step (3311), the corresponding table prefix is ​​parsed out through the SQL content under the SelectStatement tag, and the Schema name is updated to the table prefix;

[0039] The parsing of table field information read by the source component in step (3313) includes:

[0040] The SQL content of the SelectStatement tag under the XMLProperties tag is parsed. The main parsed content includes three parts: fields, table information to which the fields belong, and relative order information between fields. When the SQL content in the SelectStatement tag is empty or the field information in the SQL is an asterisk, the field information under the metadata table is queried based on the table information parsed in step (3312) or the table information read by the source component, and is used as the field information of the table read by the source component.

[0041] In some embodiments, the method for parsing the data information of the target component in step (33) includes:

[0042] (3321) Parse the database information to which the table written by the target component belongs, including: determining the connection method of the database, determining the database type, and parsing and determining the schema information of the database according to the connection method of the database;

[0043] (3322) Parse the table information written by the target component, i.e., the target table;

[0044] (3323) Parse the field information of the table written by the target component;

[0045] In step (3322), the TableName tag under the XMLProperties tag of the target component records the name of the target table. First, the TableName tag is parsed to obtain the name of the target table and the schema information of the database to which the table written by the target component belongs. If the schema information exists, it is used as the database information written by the target component. If the content of the TableName tag is empty, the name of the target table or the name of the target table and the schema information to which it belongs are parsed through the SQL tag content in the target component to complete the parsing of the table information written by the target component.

[0046] In step (3323), the field information of the table written by the target component is parsed. The DSSchema attribute value of the target tag in the target component is used to parse the field and order of the data to be written, and the field name and the relative order of the fields are parsed.

[0047] In some embodiments, the method for parsing the data information of the intermediate component in step (33) includes:

[0048] (3331) Determine the field information transmitted by the output line of the intermediate component;

[0049] (3332) Determine the field information transmitted by the input lines of the intermediate components;

[0050] (3333) Determine the correspondence between the field information transmitted by the input and output lines of the intermediate components.

[0051] The intermediate component analyzes the correspondence between the processed and unprocessed field information. Its output lines record this correspondence. Based on the output line name, the corresponding attribute value is read. If all F41 attribute values ​​are field information, the intermediate component's data operation does not involve field transformation. The data information in the intermediate component's input and output lines is consistent and represents the data parsed from the input endpoint components corresponding to all input lines of the intermediate component. If attribute value F41 contains line information, the intermediate component's data operation involves field transformation. The fields output by the intermediate component come from different input lines, and the order of the output fields matches the order of the input fields in the F41 attribute value.

[0052] In some embodiments, step (4) of constructing data lineage relationships involves generating corresponding data lineage pairs based on the parsed database, table, and field information, including:

[0053] (41) Obtain data information read by any source component;

[0054] (42) According to the data flow direction, read its corresponding endpoint. If the endpoint is a target component, directly generate the association relationship between the library and the library, the table and the field according to the field order relationship between the source component and the target component, and determine whether the metadata corresponding to the library, table and field exists. If it exists, establish the association relationship and mark the name of the job to which it belongs. If it is an intermediate component, firstly, according to the data flow direction, read the data correspondence in its output line. Secondly, update the data information of the input end saved in the output line of the intermediate component until the input data in the output line is a specific library, table and field. And recursively query until the next endpoint component is the target component, and determine whether the library, table and field information in the target component and the input target component line exists. If it exists, establish the association relationship and record the association relationship in the database lineage table.

[0055] In some embodiments, step (1) also includes periodically exporting all content under the DataStage tool offline, and obtaining all projects under the cross-system data integration tool DataStage from the exported content.

[0056] Compared with the prior art, the advantages and positive effects of the present invention are:

[0057] The cross-system metadata lineage automatic parsing method of the present invention achieves accurate parsing of four key components in the data extraction operation by automatically reading the cross-system lineage files under all projects, and completes the construction of the endpoint-line-endpoint structure based on the data flow relationship between key components, thus solving the problem of automatic adaptation and identification of flexible combinations between different components in the data extraction operation.

[0058] Secondly, based on the metadata information collected and stored by various banking systems, combined with the data flow between different components and the database, table, and field information involved in each component, the data lineage between systems, between databases, between tables, and between fields is constructed and stored.

[0059] Based on the stored data lineage, a graphical representation of the data lineage is provided to show the data lineage relationships between different metadata and the relationship between the data lineage and the corresponding lineage files. This enables rapid data lineage analysis while achieving precise location of data lineage files, facilitating the synchronous modification of changes.

[0060] Other features and advantages of the present invention will become clearer after reading the detailed embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart of an embodiment of the cross-system metadata lineage automatic resolution method proposed in this invention;

[0063] Figure 2 This is a flowchart illustrating the component data lineage parsing process in one embodiment of the cross-system metadata lineage automatic parsing method proposed in this invention.

[0064] Figure 3 This is a flowchart illustrating the parsing process of data information from the source component in one embodiment of the cross-system metadata lineage automatic parsing method proposed in this invention.

[0065] Figure 4 This is a flowchart illustrating the parsing process of data information of a target component in one embodiment of the cross-system metadata lineage automatic parsing method proposed in this invention.

[0066] Figure 5This is a flowchart illustrating the parsing process of data information from intermediate components in one embodiment of the cross-system metadata lineage automatic parsing method proposed in this invention.

[0067] Figure 6 This is a schematic diagram of the data extraction operation in one embodiment of the cross-system metadata lineage automatic resolution method proposed in this invention;

[0068] Figure 7 This is a visualization of data lineage relationships in one embodiment of the cross-system metadata lineage automatic resolution method proposed in this invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] It should be noted that in the description of this invention, terms such as "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," indicating directional or positional relationships, are based on the directional or positional relationships shown in the accompanying drawings. These are merely for ease of description and do not indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation; therefore, they should not be construed as limitations on this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0071] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0072] Example 1

[0073] This embodiment proposes a method for automatic cross-system metadata lineage resolution, such as... Figure 1 As shown, it includes:

[0074] (1) Steps for obtaining cross-system lineage files and configuration files, including:

[0075] Retrieve all projects under the cross-system data integration tool DataStage;

[0076] Retrieve all data extraction jobs under the project. The data extraction job file must record at least the data flow information of the component.

[0077] (2) The steps for component identification and relationship construction of cross-system lineage files include:

[0078] (21) Identify all components in the data sampling operation, and obtain the data flow information of the components and the relationships between the components;

[0079] (22) Based on the data flow information of the components, the types of each component are defined as source components, intermediate components and target components, and the three types of components are constructed into an endpoint-line-endpoint data structure. The source component is a component with only data outflow, the target component is a component with only data inflow, and the intermediate component is a component with both data outflow and data inflow. In the constructed endpoint-line-endpoint data structure, the endpoint is a component with data inflow or outflow, and the line records the data flow information and the data content that flows through it.

[0080] (3) Data information parsing steps, including: parsing the specific data transmitted between components, including parsing the metadata information carried by each component and the correspondence between the metadata information, wherein the metadata information includes library information, table information and field information;

[0081] (4) The steps of constructing data lineage include: sorting out the relationship between metadata based on the parsed data information, and generating corresponding data lineage pairs based on the data flow information and the metadata information parsed from the metadata information to complete the construction of data lineage.

[0082] (5) The steps for visualizing data lineage relationships include: generating wireframe diagrams of the constructed data lineage relationships and displaying them.

[0083] The cross-system metadata lineage automatic parsing method in this embodiment achieves accurate parsing of four key components in the data extraction operation by automatically reading the cross-system lineage files under all projects, and completes the construction of the endpoint-line-endpoint structure based on the data flow relationship between key components, thus solving the problem of automatic adaptation and identification of flexible combinations between different components in the data extraction operation.

[0084] Secondly, based on the metadata information collected and stored by various banking systems, combined with the data flow between different components and the database, table, and field information involved in each component, the data lineage between systems, between databases, between tables, and between fields is constructed and stored.

[0085] Based on the stored data lineage, a graphical representation of the data lineage is provided to show the data lineage relationships between different metadata and the relationship between the data lineage and the corresponding lineage files. This enables rapid data lineage analysis while achieving precise location of data lineage files, facilitating the synchronous modification of changes.

[0086] In step (2), to address the unified adaptation and parsing of data extraction operations under flexible combinations of different components, after all cross-system lineage files and configuration information have been acquired, this embodiment decompresses all cross-system files (i.e., ISX format files), and simultaneously iterates through and reads the decompressed cross-system files to construct an endpoint-line-endpoint data structure. This enables the identification, relationship construction, and storage of key component information within the data extraction operation. The specific steps include:

[0087] Define the component information under the sampling operation.

[0088] The data extraction operation content obtained in this embodiment is a composite format of a bin file. To realize the parsing of data extraction operations composed of different components, this invention abstracts and defines the different components. First, the content of the data extraction operation is abstractly defined as consisting of one or more endpoint-line-endpoint structures. Second, the endpoint components are further subdivided into source components, intermediate components, and target components. Third, for the source components, which only have output lines, the data flow is actually the data transfer between different endpoints. The data transfer is connected through different lines, realizing the identification and sequential connection of different endpoint-line-endpoint structures, thus realizing the overall description of the data flow.

[0089] In some embodiments, the method for step (21) of identifying all components in the sampling operation and obtaining data flow information of the components and the relationships between the components includes:

[0090] (211) Iterate through each sampling operation and read all endpoints in the sampling operation.

[0091] Specifically, it reads the content between the BEGIN DSRECORD code segment and the END DSRECORD code segment, which records the definitions, outputs, and output information of different components.

[0092] The identification principles for endpoint components are set. The difference between the source component, intermediate component, and target component lies in the data input and output of the component. The source component only involves data output, the intermediate component involves data input and output, and the target component only involves data input.

[0093] (212) Identify endpoint information, including identifying the name and type of the components involved in the endpoint and finding the job name.

[0094] Specifically, the system reads the different attribute information between the BEGIN DSRECORD and END DSRECORD code segments. First, it searches for the Identifier attribute to identify the component name. Second, it searches for the F1 attribute to identify the job type; for endpoint information, the value is always "STAGE". Third, it searches for the F9 and F10 component contents. If only F9 exists, it means that the component is defined as a target component. If only F10 exists, it means that the component is defined as a source component. If both F9 and F10 exist, it means that the component is defined as an intermediate component.

[0095] (213) Identify the relationship between endpoints and lines. After completing component identification, identify the relationship between components and lines.

[0096] The relationship between endpoints and lines can be identified through the definitions of F9 and F10. F9 defines how data is transmitted from a line to an endpoint, and its specific structure is: job name.endpoint name.line name. F10 records how data is transmitted from an endpoint to a line, and its specific structure is the same as F9. Parsing F9 and F10 can obtain the data input line and data output line of the specific endpoint.

[0097] At this point, the endpoint name, endpoint type, and input / output line information are obtained.

[0098] After identifying the endpoints and the relationships between endpoints and lines, in order to construct the data structure between endpoints-lines-endpoints, it is also necessary to identify different lines and the endpoint information corresponding to the lines.

[0099] In some embodiments, after identifying the specific line information, it is necessary to identify the endpoint information involved in the line. Specifically, the definition content of F16 between the BEGIN DSRECORD code segment and the END DSRECORD code segment can be read. Its content structure is: source component / target component. Parsing the content of F16 can obtain the information of the line and the involved endpoints. The method for constructing the endpoint-line-endpoint data structure in step (22) includes:

[0100] The data flow information of the line is obtained based on the line name. The data flow information includes at least the components connected to both ends of the line, and an endpoint-line-endpoint data structure is established accordingly.

[0101] At this point, the identification of all component information and the relationships between components in the data extraction operation has been achieved, and the component group name, component type, input and output information have been recorded in the database. Taking any source component as the entry point, firstly, the next endpoint component information can be determined based on the output information. Secondly, it is determined whether it is an intermediate component or a target component. If it is a target component, the data flow is completed. If it is an intermediate component, its input line and output line information are queried. This process is repeated recursively to query its source endpoint or output endpoint information until the endpoint information is a source component or a target component. At this point, the data flow of the entire data extraction operation is completed.

[0102] In step (3), the data information parsing step is necessary because the data information carried in different components is different. After the overall construction of the data flow is completed, the specific data in the data flow needs to be parsed, that is, the metadata information (library, table, field) carried by the content of different components and the correspondence between the metadata information need to be parsed.

[0103] For endpoint components, the source component mainly records the data information to be read, the intermediate component mainly records the processing of different data information, and the target component mainly records the data information to be written.

[0104] For parsing data information of source components, target components and intermediate components, this invention provides a complete data information parsing method to realize the parsing of database, table and field information in source components and target components.

[0105] In some embodiments, such as Figure 2 As shown, the method for parsing the specific data transmitted between components in step (3) includes:

[0106] (31) Locate the component based on its name.

[0107] The specific content of the data extraction task is a composite format. In this invention, the data extraction task content is traversed, and the component position is located according to the component name. The specific matching rule is: STAGE: component name.

[0108] (32) Locate the data information of the source component, target component, and intermediate component according to the type of the component. The data information of the source component is stored in the connector and XMLProperties tags, the data information of the target component is stored in the target, connector, and XMLProperties tags, and the data processing information of the intermediate component is recorded in the information of its corresponding output line.

[0109] For source component data, the main data information is stored in the connector tag and XMLProperties tag. For target component data, the main data information is stored in the target tag, connector tag, and XMLProperties tag. For intermediate component data processing information, the main data processing information is recorded in the corresponding output line information, that is, reading the data information and corresponding processing information in F41 between the BEGIN DSRECORD code segment and the END DSRECORD code segment.

[0110] (33) The data information and data processing information of the source component, target component and intermediate component are analyzed respectively.

[0111] (34) This completes the parsing of database, table and field information in the source component and target component, the parsing of the corresponding relationship of data processing in the intermediate component, and stores the information in the output lines corresponding to the source component and the intermediate component.

[0112] To establish data lineage, it is first necessary to accurately parse the database information from which the source component reads the table, i.e., the schema information to which the table belongs.

[0113] Parse and determine the schema information. The meaning of the schema information varies depending on the database type. The schema information should be obtained according to the specific database type, under the XMLProperties tag. <datasource> 、 <username>The tags record the URL of the database configuration and the basic information of the connected user, and the correspondence between the general database type and the schema information is shown in Table 1:

[0114] Database type Retrievable tag content Explanation of reasons Oracle <username> < / username> <username>The database user information was recorded, and the schema name in the Oracle database was consistent with the database user. < / username> DB2 <username> < / username> <username>The database user information for the data extraction database is recorded, and the schema name in the DB2 database is consistent with the database user. < / username> MySQL <datasource> < / datasource> <datasource>The URL information for connecting to the database is recorded, and the schema name in the MySQL database is consistent with the database name in the URL; < / datasource> Hive <datasource> < / datasource> <datasource>The URL information for connecting to the database is recorded, and the schema name in the Hive database is consistent with the database name in the URL; < / datasource> SQL Server <datasource> <username> < / username> < / datasource> SQL Server databases differ from other databases in that the name of a table is determined by both the actual database name and the user to whom the table belongs. Therefore, it is necessary to... <datasource>Retrieve database information <username>Retrieve database user information; ensure that the schema name in the SQL Server database matches the combined name of the database name and username. < / username> < / datasource> Dameng <username> < / username> <username>The database user information for the data extraction database is recorded, and the schema name in the DM database is consistent with the database user. < / username> Inceptor <datasource> < / datasource> <datasource>The URL information for connecting to the database is recorded, and the schema name in the Hive database is consistent with the database name in the URL; < / datasource>

[0115] Table 1

[0116] In some embodiments, such as Figure 3 , Figure 6 As shown, the method for parsing the data information of the source component in step (33) includes:

[0117] (3311) Parse the database information to which the table read by the source component belongs, including: determining the database connection method, determining the database type, and parsing and determining the schema information of the database according to the database connection method.

[0118] After determining the database connection method, schema parsing must be performed strictly according to the different database connection methods. The Datastage tool supports multiple source databases, and the schema meaning of the table under different database types is different. It is necessary to strictly parse the corresponding schema information according to the database type. This invention determines the database type by parsing the name attribute value under the connector tag. The composition rule of the name attribute value is the database type Connector, such as: OracleConnector, which means that the source database is an Oracle database. Parsing the name attribute value can determine the database type.

[0119] (3312) Parse the table information read by the source component.

[0120] (3313) Parse the field information of the table read by the source component.

[0121] The database connection method can be either ODBC or JDBC. This can be distinguished by parsing the name attribute value within the connector tag. A value of ODBCConnector indicates that the database connection is made via ODBC, while other values ​​indicate that the connection is made via JDBC.

[0122] In step (3311), when the database connection method is JDBC connection, there are usually two situations when parsing the content of the DataSource tag and Username tag. First, when the content under the tag is the database connection information, the tag value is read directly to parse out the corresponding schema information.

[0123] Secondly, when the content under the tag is a parameter, the specific parameter value is queried through the parameter set, and the database schema information is obtained from the specific parameter value. The parameter set can be obtained by identifying the pst file under the .isx project file. The file name of the pst file is the name of the parameter set. The file content stores the specific values ​​corresponding to the specific parameters under the parameter set. The URL connection information and user information can be obtained through the defaultValue value corresponding to the parameter under the .pst file. When the specific value corresponding to the parameter is a marker that references the environment variable, the corresponding environment variable is obtained through the project to parse the specific parameter value, that is, the specific value corresponding to the parameter name is parsed through the DSParams file.

[0124] When the database connection method is ODBC, the value of the DataSource tag is parsed to obtain the ODBC data source name. The parsing method is similar to that used for JDBC connections. After obtaining the data source name, the URL information corresponding to the data source and the connection user information need to be obtained through the ODBC configuration file .odbc.ini. In other words, by obtaining the URL information corresponding to the data source and the connection user information through the ODBC configuration file, the schema information of the database can be obtained.

[0125] The steps for resolving ODBC connection methods include: matching the ODBC data source name, which can be done by performing a full-word match in the .odbc.ini file using the corresponding tag value of DataSource.

[0126] After matching the corresponding data source name, the database type is determined by the Description attribute value under the data source. Based on the specific database type, the corresponding database information and user information can be obtained. Specifically, the Database and LogonID attribute values ​​under the .odbc.ini file can be used to obtain them. This completes the initial parsing of the schema information of the table in the source component.

[0127] In some embodiments, parsing the table information read by the source component in step (3312) includes:

[0128] The content of the SelectStatement tag under the XMLProperties tag is parsed, which records the specific database, table, and field information queried by the source component. If the SelectStatement tag is empty, it is obtained from the content of the Table tag under the XMLProperties tag. If the Table tag is empty, the table information in the source component is parsed from the SQL content under the SelectStatement tag by the SQL parser.

[0129] In addition, considering that data extraction users are usually read-only users in practice, the schema name parsed by DataSource and Username in this case is not the actual schema information of the table. When the parsed schema name is inconsistent with the actual schema name of the table, the corresponding table prefix is ​​parsed from the SQL content under the SelectStatement tag, and the schema name is updated to the table prefix.

[0130] The parsing of table field information read by the source component in step (3313) includes:

[0131] The parsing of field information in the source component information is mainly done through the SQL content of the SelectStatement tag under the XMLProperties tag. The main parsing content includes three parts: the field, the table information to which the field belongs, and the relative order information between the fields.

[0132] To address the issue of unparseable field information when the SQL content of the SelectStatement tag is empty or the field information in the SQL is an asterisk ("*"), when the SQL content of the SelectStatement tag is empty or the field information in the SQL is an asterisk, the field information under the metadata table is queried based on the metadata information and the table information read by the source component, and used as the field information of the table read by the source component.

[0133] This completes the parsing of the library, table, and field information in the source component, and stores this information in the corresponding data line of the source component.

[0134] In some embodiments, such as Figure 4 As shown, the method for parsing the data information of the target component in step (33) includes:

[0135] (3321) Parse the database information to which the table written by the target component belongs, including: determining the connection method of the database, determining the database type, and parsing and determining the schema information of the database according to the connection method of the database.

[0136] (3322) Parse the table information written by the target component, i.e., the target table;

[0137] (3323) Parse the field information of the table written by the target component;

[0138] The parsing of the table information to be written in the target component is different from the parsing method of the table in the source component. In step (3322), the TableName tag under the XMLProperties tag of the target component records the name of the target table. First, the TableName tag is parsed to obtain the name of the target table and the schema information of the database to which the table written by the target component belongs. If the schema information exists, it is used as the database information to be written by the target component. Second, if the content of the TableName tag is empty, the name of the target table or the name of the target table and the schema information to which it belongs are parsed through the SQL tag content in the target component to complete the parsing of the table information written by the target component.

[0139] In step (3323), the field information of the table written by the target component is parsed. This is done by parsing the DSSchema attribute value of the target tag within the target component, which records the fields to be written and their order, and parses the field names and relative order of the fields. This completes the parsing of the database, table, and field information to be written to the target component.

[0140] The data information parsing of intermediate components mainly includes the parsing of two parts: first, the database, table, and field information before data processing; second, the correspondence between the database, table, and field information before and after the processed data fields.

[0141] In some embodiments, such as Figure 5 As shown, the method for parsing the data information of the intermediate components in step (33) includes:

[0142] (3331) Determine the field information transmitted by the output line of the intermediate component;

[0143] (3332) Determine the field information transmitted by the input lines of the intermediate components;

[0144] (3333) Determine the correspondence between the field information transmitted by the input and output lines of the intermediate components.

[0145] The intermediate component analyzes the correspondence between the processed and unprocessed field information. The output lines record this correspondence. The actual input information received by the intermediate component can be determined by reading the attribute value of F41 between the BEGIN DSRECORD and END DSRECORD code segments corresponding to the output line name. If the attribute values ​​are all field information, the intermediate component's data operation does not involve field transformation; the data information in the input and output lines is consistent and represents the data parsed from the input endpoint components corresponding to all input lines of the intermediate component. If the attribute values ​​include line information, the intermediate component's data operation involves field transformation; the fields output by the intermediate component come from different input lines and maintain consistency with the field information and order of the parsed lines' corresponding input endpoint components.

[0146] This completes the data parsing of the intermediate components and records the information in the corresponding data lines of the intermediate components.

[0147] In this embodiment, the construction of data lineage is based on real metadata from various bank systems. At the same time, based on the parsed data information, the association between corresponding metadata and the generation and construction of data lineage are completed.

[0148] In the actual parsing and construction process, this invention uniformly collects and stores metadata information under various system databases. The collection method is to connect to the database via JDBC and use the conn.getMetaData method to obtain metadata information under different database types. The specific metadata information collected includes schema, table, and field information, and is uniformly stored according to the relationship between schema and table, and table and field, and information of relevant logical systems is added.

[0149] In some embodiments, step (4) of constructing data lineage relationships involves generating corresponding data lineage pairs based on the parsed database, table, and field information, including:

[0150] (41) Obtain data information read by any source component.

[0151] (42) According to the data flow direction, read its corresponding endpoint. If the endpoint is the target component, directly generate the association relationship between the library and the library, the table and the field according to the field order relationship between the source component and the target component, and determine whether the metadata corresponding to the library, table and field exists. If it exists, establish the association relationship and mark the name of the job to which it belongs. If it is an intermediate component, firstly, according to the data flow direction, read the data correspondence in its output line. Secondly, update the data information of the input end saved in the output line of the intermediate component, that is, replace the line information in the original data information with the specific library and table information until the input data in the output line is the specific library, table and field. And recursively query until the next endpoint component is the target component, and determine whether the library, table and field information in the target component and the input target component line exists. If it exists, establish the association relationship and record the association relationship in the database lineage table.

[0152] like Figure 7 As shown, in step (5), the visualization of data lineage is mainly based on the associations stored in the lineage table of the database to graphically display the lineage relationship, so as to facilitate the understanding and use of the lineage by users.

[0153] The invention specifically demonstrates information about the data source and user, namely: logical system, database, table, and field information. It also demonstrates the lineage file system involved in specific lineage relationships. Specifically, by reading the metadata information collected and stored in the database, it achieves a comprehensive display of data lineage and solves the problem of unclear scope of impact and unclear location of impact on downstream systems when upstream business system databases are changed during actual development.

[0154] To address the issue of automatically acquiring massive amounts of cross-system lineage files and reduce the impact of frequent queries on the DataStage data integration tool server during lineage resolution on data extraction performance, this embodiment employs a periodic offline export method to achieve rapid acquisition of all project content under the DataStage tool. In some embodiments, step (1) further includes periodically exporting all content under the DataStage tool offline, and obtaining all projects under the cross-system data integration tool DataStage from the exported content. The time period is specifically configured in conjunction with the change cycle of the data extraction operation under the enterprise's DataStage data integration tool to ensure the timely updating of the data extraction operation content and the timeliness of the lineage resolution results. After comparing different solutions, this embodiment designs an offline export method, namely, deploying an automated script project to automatically obtain the lineage files between systems, instead of using the method of accessing the cross-system data extraction tool DataStage server in real time to obtain the project files. It periodically obtains all project files, environment variable configuration files, and configuration files connected via ODBC within a set time interval and stores them in a specified server path. This effectively solves the resource consumption impact caused by frequent access to the cross-system data extraction tool DataStage server resources during real-time reading and parsing.

[0155] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions claimed by the present invention.< / username> < / datasource>

Claims

1. A cross-system metadata lineage automatic resolution method, characterized in that, Comprise: (1) Cross-system blood file and configuration file acquisition step, comprising: Acquire all projects under the cross-system data integration tool DataStage; Acquire all throw jobs under the project, and at least record the data flow information of the components in the file of the throw job; (2) Cross-system blood file component identification and relationship construction step, comprising: (21) Identify all components in the throw job, and acquire the data flow information of the components and the relationship between the components; (22) According to the data flow information of the components, define the types of the components, respectively, as source components, intermediate components, and target components, and construct a data structure of endpoint-line-endpoint for the three types of components, the source component is a component with only data outflow, the target component is a component with only data inflow, and the intermediate component is a component with both data outflow and data inflow, in the constructed data structure of endpoint-line-endpoint, the endpoint is a component with data inflow or outflow, and the line records data flow information and data content flowing through it; (3) Data information analysis step, comprising: analyzing the specific data transmitted between components, including respectively analyzing the metadata information carried by each component and the corresponding relationship between the metadata information, the metadata information includes library information, table information and field information; (4) Constructing data blood relationship step, comprising: according to the analyzed data information, combing the association relationship between the metadata, and according to the data flow information, generating corresponding data blood relationship pairs according to the metadata information, and completing the construction of data blood relationship; (5) Visual display step of data blood relationship, comprising: generating a wireframe diagram of the constructed data blood relationship and displaying it; The method of analyzing the specific data transmitted between components in step (3) comprises: (31) According to the name of the component, locate the position of the component; (32) According to the type of the component, respectively locate the data information position of the source component, the target component, and the intermediate component, wherein the source component data information is stored in the connector, XMLProperties tag, the target component data information is stored in the target, connector, XMLProperties tag, and the data processing information of the intermediate component is recorded in the information of the output line corresponding to it; (33) Analyze the data information of the source component, the target component, and the intermediate component respectively; (34) At this point, the database, table, and field information in the source component and the target component are analyzed, the intermediate component data processing corresponding relationship is analyzed, and the information is stored in the output line corresponding to the source component and the intermediate component.

2. The cross-system metadata bloodline automatic resolution method of claim 1, wherein, The method of identifying all components in the throw job and acquiring the data flow information of the components and the relationship between the components in step (21) comprises: (211) Loop through each throw job respectively, read all endpoints in the throw job; (212) Identify endpoint information, including identifying the name and type of the component involved by the endpoint, and finding the job name; (213), identifying the relationship between the end point and the line, after the component identification, the component and line relationship identification is performed, and the structure is specifically defined as: job name.component name.line name.

3. The cross-system metadata bloodline automatic resolution method of claim 2, wherein, The method for constructing the end point-line-end point data structure in step (22) comprises: According to the line name, the data flow direction information of the line is obtained, the data flow direction information at least includes the components connected at both ends of the line, and the end point-line-end point data structure is established according to the data flow direction information.

4. The cross-system metadata bloodline automatic resolution method of claim 1, wherein, The method for parsing the data information of the source end component in step (33) comprises: (3311), parsing the information of the database to which the table read by the source end component belongs, including: determining the connection mode of the database, determining the database type, and determining the Schema information of the database according to the connection mode of the database; (3312), parsing the table information read by the source end component; (3313), parsing the field information of the table read by the source end component; In step (3311), when the connection mode of the database is JDBC connection, the contents of the DataSource tag and the Username tag are parsed, when the contents under the tag are the connection information of the database, the tag value is directly read to parse the corresponding Schema information; When the contents under the tag are parameters, the specific parameter values are queried through the parameter set, and the Schema information of the database is obtained from the specific parameter values, the parameter set can be obtained by identifying the pst file under the isx engineering file, and the file name of the pst file is the name of the parameter set, when the specific value corresponding to the parameter is a mark of a reference environment variable, the corresponding environment variable is obtained through the engineering to parse the specific parameter value; When the connection mode of the database is ODBC connection, the tag value of the DataSource tag is parsed, the data source name of ODBC is obtained, the URL information and the connection user information corresponding to the data source are obtained through the ODBC configuration file, and the Schema information of the database is obtained.

5. The cross-system metadata bloodline automatic parsing method according to claim 4, wherein In step (3312), the table information read by the source end component is parsed, including: The content of the SelectStatement tag under the XMLProperties tag is parsed, which records the specific database, table and field information queried by the source end component, if the SelectStatement tag is empty, the Table tag content under the XMLProperties tag is obtained, if the Table tag is empty, the table information in the source end component in the SQL content under the SelectStatement tag is parsed through the SQL parsing program; When the parsed Schema name is inconsistent with the actual table Schema name, the corresponding table prefix is parsed through the SQL content under the SelectStatement tag, and the Schema name is updated to the table prefix; The field information of the table read by the source component in step (3313) includes: The SQL content of the SelectStatement tag under the XMLProperties tag is parsed, and the main parsed content includes three parts: fields, table information to which the fields belong, and relative order information between the fields; when the SQL content in the SelectStatement tag is empty or the field information in the SQL is a star, the field information in the metadata table is queried based on the metadata information and the table information read by the source component, and is used as the field information of the table read by the source component.

6. The cross-system metadata bloodline automatic resolution method of claim 1, wherein, The method for parsing the data information of the target component in step (33) includes: (3321), parsing the database information to which the table written by the target component belongs, including: determining the connection mode of the database, determining the database type, and determining the Schema information of the database according to the connection mode of the database; (3322), parsing the table information written by the target component, that is, the target table; (3323), parsing the field information of the table written by the target component; In step (3322), the TableName tag under the XMLProperties tag of the target component records the name of the target table. First, the name of the target table and the Schema information of the database to which the table written by the target component belongs are obtained by parsing the TableName tag, and if the Schema information exists, the Schema information is used as the write-in database information of the target component; if the content of the TableName tag is empty, the target table name or the target table name and the Schema information to which the target table belongs are parsed through the SQL tag content in the target component, and the parsing of the table information written by the target component is completed. In step (3323), the field information of the table written by the target component is parsed, and the DSSchema attribute value of the target tag in the target component is parsed, which records the fields and the order of the data to be written, and the field name and the relative order of the fields are parsed.

7. The cross-system metadata bloodline automatic resolution method of claim 1, wherein, The method for parsing the data information of the intermediate component in step (33) includes: (3331), determining the field information transmitted by the output line of the intermediate component; (3332), determining the field information transmitted by the input line of the intermediate component; (3333), determining the corresponding relationship between the field information transmitted by the input line and the output line of the intermediate component.

8. The cross-system metadata lineage automatic resolution method of claim 7, wherein, In step (4) of constructing the data blood relationship, the corresponding data blood relationship pairs are generated according to the parsed database, table and field, including: (41), obtaining the data information read by any source component; (42), according to the data flow, read its corresponding endpoint, if the endpoint is the target component, then directly according to the field order relationship between the source end component and the target component to generate the association relationship of library and library, table and table, field and field, and judge whether the metadata corresponding to the library, table and field exists, if it exists, establish the association relationship, and mark the name of the job it belongs to; If it is an intermediate component, first, according to the data flow, read the data corresponding relationship in the output line, second, update the data information of the input end saved by the output line of the intermediate component, until the input data in the output line is specific library, table and field, and recursively query with this until the next endpoint component is the target component, and judge whether the target component and the library, table and field information in the input target component line exist, if they exist, establish the association relationship, and record the association relationship under the database blood relationship table.

9. The cross-system metadata lineage automatic resolution method of any one of claims 1-8, wherein, In step (1), it also includes periodically exporting all contents under the DataStage tool offline, obtaining all projects under the cross-system data integration tool DataStage from the exported contents.

Images