Data blood relationship construction method and device, display system, medium and equipment

Abstract

The invention provides a data blood relationship construction method and device, a display system, a medium and equipment, and is applied to the technical field of data management. According to the method, description information of a plurality of job streams is obtained, each job stream comprises a plurality of execution nodes, each execution node is provided with a node running path, and the description information comprises codes and configuration data related to the job streams; determining a scene category to which each job stream belongs by utilizing the description information; and for each job flow under any scene category, analyzing the relationship among the job flow, the node and the table, and constructing the relationship among the job flows, the relationship between the job flow and the node, the relationship among the nodes, the relationship between the node and the table, the relationship among the tables, the relationship between the table and the view and the relationship among the fields through the dependency event, the node running path diagram and the SQL statement analysis. According to the method, by analyzing the description information of the workflow and the nodes of the workflow, various relationships among the data hierarchies can be comprehensively constructed, so that the accuracy and depth of data blood relationship analysis are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to a method, device, display system, medium and equipment for constructing data blood relationship. Background Art

[0002] With the advancement of information technology, data-intensive industries such as banks face huge data management challenges, especially in the processing of historical data and duplicate data, which puts pressure on storage and computing resources and affects the effectiveness of data assets. In terms of risk control and anti-fraud, banks also need advanced methods to identify and prevent risks.

[0003] Data lineage research has become the key to improving data governance. It helps banks track data changes, evaluate data credibility, and optimize storage and use by revealing the source, flow, and evolution of data. However, traditional methods are difficult to meet multiple levels of needs at the same time, and usually use a graph structure, which is not intuitive enough in the face of big data environments.

[0004] Therefore, how to more comprehensively analyze and display data lineage relationships has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of the above problems, the present invention provides a data lineage relationship construction method, device, display system, medium and equipment that overcome the above problems or at least partially solve the above problems. The technical solution is as follows:

[0006] A method for constructing data blood relationship, comprising:

[0007] Obtaining description information of a plurality of job flows, wherein the job flows include a plurality of execution nodes, each of the execution nodes is provided with a node running path, and the description information includes code and configuration data related to the job flows;

[0008] Using the description information, determining the scene category to which each of the job flows belongs;

[0009] For each of the job flows under any scenario category: parse the relationship between job flows, nodes, and tables, and build the relationship between job flows, the relationship between job flows and nodes, the relationship between nodes, the relationship between nodes and tables, the relationship between tables, the relationship between tables and views, and the relationship between fields through dependency events, node operation path diagrams, and SQL statement analysis.

[0010] Optionally, the analysis of dependency events, node operation path diagrams and SQL statements to construct relationships between job flows, relationships between job flows and nodes, relationships between nodes, relationships between nodes and tables, relationships between tables, relationships between tables and views, and relationships between fields includes:

[0011] According to the dependency events in the configuration data of each of the operation flows, a first-level data lineage relationship between the operation flows is established;

[0012] respectively constructing the second-layer data kinship relationship between each of the operation flows and each of the execution nodes contained therein;

[0013] For the same job flow: constructing a third-layer data kinship relationship between the execution nodes according to the node running paths of the execution nodes included in the job flow;

[0014] Performing logical judgment according to the code content of each execution node to confirm whether a node table exists for each execution node;

[0015] For the execution node having the node table: constructing a fourth-layer data lineage relationship between the execution node and its node table;

[0016] According to the preceding codes of the node tables in the corresponding code content, a fifth layer of data kinship relationship between the node tables is constructed;

[0017] According to the view construction statements of each node table in the corresponding code content, a sixth layer of data kinship relationship between the node table and the view is constructed;

[0018] For any of the node tables: according to the execution statements of the node table in the corresponding code content, the seventh layer of data kinship relationship between the fields in the node table is constructed.

[0019] Optionally, the code and configuration data related to the job flow include basic information, input and output dependencies, historical operation data, database table information and attributes of the job flow.

[0020] Optionally, after constructing the relationship between job flows, the relationship between job flows and nodes, the relationship between nodes, the relationship between nodes and tables, the relationship between tables, the relationship between tables and views, and the relationship between fields, the method further includes:

[0021] According to the relationship between the job flows, the relationship between the job flows and the nodes, the relationship between the nodes, the relationship between the nodes and the tables, the relationship between the tables, the relationship between the tables and the views, and the relationship between the fields, a lineage dimension table and other data tables storing subject attribute information are constructed.

[0022] Optionally, the other data tables storing subject attribute information include at least one of a job flow information table, a table / view information table, a job node information table and a field information table.

[0023] Optionally, the method further includes:

[0024] Obtaining a target data entity to be displayed, wherein the data entity includes at least one entity of a scene, a job flow, a job, a data table, a view, and a field;

[0025] Based on the lineage dimension table related to the target data entity, the data lineage relationship between the target data entity and other data entities is displayed in a graphical form.

[0026] A data kinship relationship building device comprises: a workflow description information obtaining unit, a scene category determining unit and a relationship parsing unit.

[0027] The job flow description information obtaining unit is used to obtain description information of a plurality of job flows, wherein the job flow includes a plurality of execution nodes, each of the execution nodes is provided with a node running path, and the description information includes code and configuration data related to the job flow;

[0028] The scene category determination unit is used to determine the scene category to which each of the job flows belongs by using the description information;

[0029] The relationship parsing unit is used to parse the relationships among job flows, nodes, and tables for each job flow under any scenario category, and to construct relationships among job flows, job flow and node, node and table, table and view, and field through dependency events, node operation path diagram, and SQL statement analysis.

[0030] A data kinship display system includes: a graphical display module, a search module and an interactive module.

[0031] The graphical display module is used to display the data kinship relationship of data entities in the form of diagrams;

[0032] The search module is used to search through the blood relationship dimension table according to the search conditions and return a result list;

[0033] The interaction module is used to support the user to click on the data entity and its connection line to expand the data lineage relationship at the next level.

[0034] A computer-readable storage medium stores a program, which, when executed by a processor, implements any of the above-mentioned methods for establishing data bloodline relationships.

[0035] An electronic device, comprising at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; and the processor is used to call program instructions in the memory to execute any of the data blood relationship construction methods described above.

[0036] By means of the above technical scheme, the data lineage relationship construction method, device, display system, medium and equipment provided by the present invention include: obtaining description information of multiple job flows, wherein the job flow includes multiple execution nodes, each execution node is provided with a node running path, and the description information includes code and configuration data related to the job flow; using the description information, determining the scene category to which each job flow belongs; for each job flow under any scene category: parsing the relationship between the job flow, node, and table, and constructing the relationship between job flows, the relationship between job flows and nodes, the relationship between nodes, the relationship between nodes and tables, the relationship between tables, the relationship between tables and views, and the relationship between fields through dependency events, node running path diagrams and SQL statement analysis. The present invention can comprehensively construct multiple relationships between data hierarchies by parsing and analyzing the description information of the job flow and its nodes, thereby enhancing the accuracy and depth of data lineage relationship analysis.

[0037] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0039] Figure 1 A schematic diagram showing a flow chart of an implementation of a method for establishing a data blood relationship provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram showing a system list of a data lineage relationship display system provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram showing a data lineage relationship page of a data lineage relationship display system provided by an embodiment of the present invention

[0042] Figure 4 A schematic diagram of a solid line floating frame of a data lineage relationship page provided by an embodiment of the present invention is shown;

[0043] Figure 5 A schematic diagram showing a dotted floating frame of a data lineage relationship page provided by an embodiment of the present invention;

[0044] Figure 6A schematic diagram of a floating table frame of a data lineage relationship page provided by an embodiment of the present invention is shown;

[0045] Figure 7 A schematic diagram of a double solid line floating frame of a data lineage relationship page provided by an embodiment of the present invention is shown;

[0046] Figure 8 A schematic diagram showing a job flow page of a data lineage relationship display system provided by an embodiment of the present invention is shown;

[0047] Fig. 9 A schematic diagram showing a table structure page of table details of a data lineage relationship display system provided by an embodiment of the present invention is shown;

[0048] Fig.10 A schematic diagram showing the structure of a data blood relationship construction device provided by an embodiment of the present invention is shown;

[0049] Fig.11 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0050] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0051] With the continuous advancement of information technology and the explosive growth of data volume, traditional data management methods are facing unprecedented challenges. Especially in data-intensive industries such as banking, the amount of data has increased significantly in recent years, bringing tremendous pressure and challenges to data management and analysis.

[0052] The banking industry usually needs to deal with a large amount of historical data and duplicate data, which not only consumes a lot of storage and computing resources, but also affects the effectiveness and practicality of data assets. In addition, banks face urgent needs in areas such as risk control and anti-fraud, and need to use more advanced methods to identify and prevent potential risks.

[0053] In this context, the research and application of data lineage is particularly important. Data lineage can help banks identify the source, flow and evolution of data, thereby improving data governance and enhancing the understanding and management of data. By building data lineage, banks can better track data changes, evaluate the credibility and validity of data, and optimize the use and storage of data.

[0054] However, traditional data lineage methods have some limitations and are difficult to meet the needs of different levels at the same time: data lineage at the system level, program level, table level, and field level often lacks a unified and systematic solution. In addition, traditional methods usually use a graph structure to describe the relationship between data, which is not intuitive and flexible enough when facing big data and distributed computing environments.

[0055] To solve these problems, a new technical route has been proposed. By collecting job flow data in a big data cloud environment, recording its dependencies, and tracking data changes and flows, banks can more comprehensively analyze the source and use of data. In addition, combining data lineage with application scenarios can quickly locate job flows in specific scenarios, avoiding duplicate development and waste of resources.

[0056] In general, exploring and applying multi-level and diversified data lineage relationships can not only simplify complex data relationships and enable users to intuitively understand data flow and workflow operation, but also significantly improve data service quality and management efficiency.

[0057] Based on this, in an embodiment of the present invention, a method for constructing data lineage relationships is provided. By collecting and analyzing the dependencies of data workflows and their components (such as nodes, tables, views, and fields), and by parsing code and configuration data, a comprehensive data lineage relationship is constructed. Finally, these relationships are displayed in the system with a visual and interactive interface, and powerful search and navigation functions are provided to help users efficiently understand and manage data processing processes.

[0058] like Figure 1 As shown, a flow chart of an implementation of a method for establishing a data blood relationship provided by an embodiment of the present invention is provided. The method may include:

[0059] S100: Obtain description information of a plurality of job flows, wherein the job flow includes a plurality of execution nodes, each execution node is provided with a node running path, and the description information includes codes and configuration data related to the job flow.

[0060] Among them, a job flow is a collection of a series of job tasks, which are executed in a specific order to achieve specific data processing or computing goals.

[0061] The description information is detailed information about the job flow, including its structure, dependencies between tasks, and how to execute. It can also include process descriptions, configuration files, and execution environment.

[0062] Among them, the execution node is the basic unit or task in the job flow, and each node represents a specific operation or step, such as data reading, conversion, or loading.

[0063] Among them, the node running path: the node running path refers to a directed acyclic graph (DAG), which is used to represent the dependencies and execution order between execution nodes. In the job flow, DAG ensures that tasks are executed in the correct order without circular dependencies.

[0064] The codes provided in the embodiments of the present invention are program instructions for defining and implementing various operation steps in a workflow, and include logic and algorithms for actually executing tasks.

[0065] Configuration data refers to the parameters and settings used to control the execution of a job flow. These data define the operating environment and conditions of the job flow, such as database connection information, file paths, and other adjustable parameters.

[0066] Optionally, the job flow related code and configuration data include basic information of the job flow, input and output dependencies, historical operation data, database table information and properties.

[0067] The basic information refers to the general information about the job flow, including metadata such as the name, creator, creation time, version, and description of the job flow.

[0068] Among them, input-output dependency refers to the data transfer relationship between various execution nodes in the job flow. Input dependency is the data source required for the execution of a node, and output dependency is the data output generated after the execution of the node.

[0069] The historical operation data refers to the records and logs of the past execution process of the job flow, which may include the number of executions of the job flow, the records of success and failure, the running time and resource consumption, etc.

[0070] The database table information involves detailed information of the database table used in the job flow, including the table name, table structure, field name, data type, etc.

[0071] The attributes of a job flow refer to various configuration parameters and settings related to the job flow, which may include runtime parameters, environment variables, and system configuration of the job flow.

[0072] S110: Determine the scene category to which each workflow belongs by using the description information.

[0073] The scenario category is a scenario attribute classified according to the business function or application scenario of the workflow.

[0074] Specifically, an embodiment of the present invention can extract description information and annotation text in its code from a job flow. The annotation text may provide a description similar to "corporate mobile banking report data" and a detailed description containing remark information such as "deposit time point". Next, semantic similarity analysis technology is used to evaluate the similarity between the extracted job flow texts or keywords, including comparing the job flows and their related texts to determine their degree of semantic similarity. Based on the analysis results, job flows with high similarity are classified into the same category. This process helps identify and organize the scenario categories to which different job flows belong in the business, such as: corporate mobile banking, personal debit cards and other business areas or specific platforms. Job flows belonging to the same scenario category usually mean that they implement relevant data functions of the business line or provide data support for the same platform. This classification helps to quickly retrieve and locate related job flows through the scenario search function on the data management and display pages.

[0075] S120. For each job flow under any scenario category: parse the relationship between job flows, nodes, and tables, and build relationships between job flows, job flow and node, node, node and table, table, table and view, and field through dependency events, node operation path diagram, and SQL statement analysis.

[0076] Among them, dependency events refer to the mechanism that ensures that there are sufficient data conditions before starting during the operation of job flows and nodes. This usually involves configuring information such as running frequency and running time, but the key is to set the dependency between job flows. For example: if job flow A is responsible for obtaining the customer's deposit details, and job flow B is responsible for summarizing the deposit situation in all cities in the province, then the operation of B must wait for A's data for the day to be processed before it can start, so as to avoid empty results. Therefore, when configuring job flow B, you need to clearly specify its dependency on job flow A. Similarly, dependencies can also be set between nodes, but this dependency is mainly achieved through configuration at the job flow level.

[0077] Specifically, the embodiment of the present invention can extract all job flows and their related metadata information from the job flow management system, including job flow nodes, dependencies and trigger conditions, and understand the trigger and dependency relationships between job flows by analyzing the scheduling, trigger events and dependency events of the job flows.

[0078] Specifically, the embodiment of the present invention can generate a node operation path diagram for each job flow, describing the execution order and logical relationship of the nodes, and analyze the node operation path diagram to determine the direct or indirect dependency relationship between the nodes.

[0079] Specifically, the embodiment of the present invention can parse the SQL statements used in the node, identify the input and output tables involved by statically analyzing the FROM, JOIN, INSERT and other clauses in the SQL, and further parse the selection, calculation and transformation relationships of the fields in the SQL statements to understand the dependencies between the fields.

[0080] Specifically, the embodiment of the present invention can identify the relationship between tables through the join operation in the SQL statement, determine which tables are often used together or depend on the same data source, parse the view definition SQL, and determine the relationship between the view and the basic table.

[0081] Specifically, the embodiments of the present invention can construct a relationship network between various data entities (job flows, nodes, tables, views, and fields) by parsing the relationships between job flows, job flow and node, node and node, node and table, table and view, and field, and then use graphical tools (such as graphical databases or visualization software) to display these relationship networks, helping users to intuitively query and understand complex dependency relationships.

[0082] The data lineage relationship construction method provided by the present invention comprises: obtaining description information of multiple job flows, wherein the job flow comprises multiple execution nodes, each execution node is provided with a node running path, and the description information comprises code and configuration data related to the job flow; using the description information, determining the scene category to which each job flow belongs; for each job flow under any scene category: parsing the relationship between the job flow, the node, and the table, and constructing the relationship between job flows, the relationship between job flows and nodes, the relationship between nodes, the relationship between nodes and tables, the relationship between tables, the relationship between tables and views, and the relationship between fields through dependency events, node running path diagrams, and SQL statement analysis. The present invention can comprehensively construct multiple relationships between data hierarchies by parsing and analyzing the description information of the job flow and its nodes, thereby enhancing the accuracy and depth of data lineage relationship analysis.

[0083] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, another optional embodiment provided by the embodiment of the present invention, by analyzing dependent events, node operation path diagrams and SQL statements, constructing relationships between job flows, relationships between job flows and nodes, relationships between nodes, relationships between nodes and tables, relationships between tables, relationships between tables and views, and relationships between fields, may specifically include:

[0084] According to the dependency events in the configuration data of each job flow, the first layer of data lineage relationship between each job flow is constructed.

[0085] Specifically, the embodiment of the present invention can analyze the configuration of each job flow and identify the dependent events defined therein. Based on these dependent events, the upstream and downstream relationships of the job flows are determined: the job flow containing the dependent events in the configuration is regarded as the downstream job flow, and the dependent and referenced job flow is regarded as the upstream job flow, thereby accurately determining the mutual dependency relationship between different job flows.

[0086] Construct the second-layer data lineage relationship between each job flow and each execution node it contains.

[0087] Specifically, the embodiment of the present invention can construct a second layer of data lineage relationship by analyzing the inclusion relationship between the job flow and the execution nodes it contains, that is, the job flow is regarded as the upstream, and the various execution nodes contained therein are regarded as downstream nodes, so as to accurately determine the hierarchical relationship and dependency between the job flow and its execution nodes.

[0088] For the same job flow: based on the node running path of each execution node included in the job flow, the third-level data lineage relationship between each execution node is constructed.

[0089] In the same job flow, the embodiment of the present invention can establish a third-level data lineage relationship by analyzing the node running path of the execution node. The third-level data lineage relationship is based on the directed acyclic graph (DAG) of the job flow. In the directed acyclic graph, the direction of the arrow determines the execution order of the nodes: the node pointed by the arrow is downstream, and the node where the arrow starts is upstream, which describes the execution order between the nodes. Between different job flows, the hierarchical relationship of the nodes is judged based on the dependency relationship of the nodes, that is, the nodes in the dependency relationship are regarded as downstream, and the dependent nodes are regarded as upstream, thereby ensuring that the construction of the lineage relationship between the execution nodes can accurately reflect the dependency and execution order between the nodes.

[0090] Perform logical judgment based on the code content of each execution node to confirm whether a node table exists for each execution node.

[0091] For an execution node with a node table: construct a fourth-level data lineage relationship between the execution node and its node table.

[0092] Specifically, the embodiment of the present invention can analyze the code content of each execution node and perform logical judgment to confirm whether each execution node involves a node table. For those execution nodes that do have a node table, a fourth layer of data lineage relationship is further constructed. For example: in the SQL code, if a table appears after an "insert", "delete" or "truncate" statement, then the table is regarded as a result table and is downstream, while the node that executes the operation is upstream, which means that the node updates the table after execution. On the contrary, if a table appears after a "from" or "join" statement, the table is regarded as an upstream table, which indicates that the node table depends on these upstream tables for data processing or updating. In this way, the present invention can accurately reflect the relationship between an execution node and its node table.

[0093] According to the preceding codes of each node table in the corresponding code content, the fifth layer of data kinship relationship between the node tables is constructed.

[0094] Specifically, the embodiment of the present invention can determine the upstream and downstream relationship between tables through the logic of SQL code in the same MPP operator or Python operator node. For example, the table following the "insert", "delete" or "truncate" statement is considered as the result table, that is, it is downstream; and the table following the "from" or "join" statement is the source table, that is, the upstream table.

[0095] Specifically, in the embodiment of the present invention, for the Spark synchronization node, since this is an important component of cross-database data transmission, the upstream and downstream relationship can be determined according to the direction of data flow through its configuration information, including the database type, table name, fields, etc. of the source table, and the database type, table name, fields, etc. of the flow table: the source table is regarded as the upstream, and the flow table is regarded as the downstream, so as to accurately determine the data lineage relationship between tables.

[0096] According to the view construction statements of each node table in the corresponding code content, the sixth layer of data kinship relationship between the node table and the view is constructed.

[0097] Specifically, the embodiment of the present invention can determine the upstream and downstream relationship between the table and the view by checking the SQL statement for establishing the view, thereby accurately establishing the data lineage relationship between the node table and the view. For example, the view creation statement usually adopts the following basic format: "CREATE VIEW view name AS SELECT column name 1, column name 2, ... FROM table name WHERE [condition]". In this case, the view immediately following "CREATE VIEW" is regarded as the downstream, and the table appearing after "FROM" or "JOIN" is regarded as the upstream.

[0098] For any node table: according to the execution statements of the node table in the corresponding code content, the seventh layer of data kinship relationship between the fields in the node table is constructed.

[0099] Specifically, the embodiment of the present invention can parse the execution statements in the code to identify the direct mapping relationship between fields, and then recursively trace the source and destination of these fields to establish a field lineage chain, thereby accurately determining the data lineage relationship between the fields of the node table.

[0100] For example, in the case of "INSERT INTO a(a1, a2, a3, ...) SELECT b.b1, b.b2, b.b3, ...FROM b", the upstream and downstream relationship between the fields is determined by parsing the order correspondence between the sequences "[a1, a2, a3, ...]" and "[b.b1, b.b2, b.b3, ...]", that is, the field "a.a1" is downstream of the field "b.b1".

[0101] The embodiments of the present invention can clearly track the paths and dependencies of data flow by constructing data lineage relationships layer by layer, thereby improving the transparency and accuracy of data governance, which not only improves the efficiency of data processing, but also enhances the capabilities of subsequent data operation and maintenance and problem diagnosis.

[0102] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the embodiment of the present invention, after building the relationship between job flows, the relationship between job flows and nodes, the relationship between nodes, the relationship between nodes and tables, the relationship between tables, the relationship between tables and views, and the relationship between fields, the method further includes:

[0103] Based on the relationships between job flows, job flows and nodes, nodes, nodes and tables, tables, tables and views, and fields, we build lineage dimension tables and other data tables that store subject attribute information.

[0104] Among them, the blood dimension table is used to store the upstream and downstream relationships of various subjects, while other information data tables record the attribute information of each subject, thereby providing more auxiliary information and support for the system.

[0105] The field information contained in the lineage dimension table may include: upstream ID or table name, downstream ID or table name, lineage judgment type, upstream type and downstream type.

[0106] The upstream ID or table name is used to identify the unique identifier of the upstream data entity or its table name, and refers to the data object located at the beginning or earlier position in the data flow process.

[0107] The downstream ID or table name is used to identify the unique identifier of the downstream data entity or its table name, representing the entity that receives or depends on the upstream data in the direction of data flow.

[0108] The lineage judgment type is used to describe the type or nature of the relationship between upstream and downstream data entities, for example, how data is transferred from upstream to downstream, or what operation leads to this relationship.

[0109] The upstream type is used to define the type of the upstream entity to more accurately describe the nature of the upstream data.

[0110] The downstream type is used to define the type of the downstream entity to more accurately describe the nature of the downstream data.

[0111] The embodiments of the present invention can comprehensively capture and track the flow and dependency paths of data in the system by comprehensively considering multi-level relationships in the construction of data lineage relationships, so as to establish a detailed lineage dimension table and extend it to the attribute information data tables of other storage entities, thereby helping to improve data governance capabilities and identify potential problems in data flow, so as to support more efficient data management, problem diagnosis and compliance review.

[0112] Optionally, other data tables storing subject attribute information include at least one of a job flow information table, a table / view information table, a job node information table, and a field information table.

[0113] The job flow information table records the job flow information in the data processing process. The fields in the job flow information table may include tenant id, job flow id, job flow name, responsible person, description, cloud path, latest run date, number of citations and dependency mode. The information recorded in the job flow information table helps users understand and manage the overall process of data processing, as well as the scheduling and dependency relationships between various job flows.

[0114] The table / view information table stores detailed information about tables and views in the database. Fields in the table / view information table may include the table English name, table Chinese name, table size, type, whether the table retains history, latest date, whether the table is a source job, cross-database table name, filter conditions for syncing source tables, number of references, and view creation SQL, etc. The information recorded in the table / view information table helps users understand the storage structure and access method of data, especially when cross-database queries are involved.

[0115] The job node information table describes the information of each node in the job flow in detail. The fields in the job node information table may include tenant id, job node id, job node name, job flow id, data source, job type, write mode, job filter, synchronization field and result table. The information recorded in the job node information table helps users understand the specific steps of data processing and the data flow between nodes.

[0116] The field information table provides detailed information about the fields of each data table. The fields in the field information table may include the source table name, field English name, field Chinese name, field type, length, primary key flag, distribution key flag, association code, association key times, value range, and field sensitivity level. The information recorded in the field information table helps users understand the details of the data, the purpose of the field, and the mapping relationship of the field during data conversion and integration.

[0117] Optionally, an embodiment of the present invention can also obtain a target data entity to be displayed, wherein the data entity includes at least one entity of a scene, a job flow, a job, a data table, a view and a field; and based on a lineage dimension table related to the target data entity, display the data lineage relationship between the target data entity and other data entities in the form of a chart.

[0118] In banking and financial business scenarios, users can use the search function to obtain relevant target data entities, such as job flows or data tables, and use the lineage dimension table to display the lineage relationship between data entities. For example, business personnel can click on the name of the job flow to view the associated lineage map, and fully understand the work progress and strategic planning of the financial business. Technical personnel can track the lineage of tables and fields, confirm the accuracy of code references and development, and share data results. Managers can use filter items to monitor the completion and exceptions of employee job flows. The data lineage diagram shows the relationship between jobs and tables through solid and dotted lines, helping different roles to clearly understand data flows and dependencies.

[0119] The embodiment of the present invention can effectively display data lineage relationships by obtaining the target data entity to be displayed and using the lineage dimension table related thereto, and can also more intuitively display the interdependence and influence between entities such as scenes, job flows, jobs, data tables, views and fields. This not only helps business personnel quickly identify the path and potential bottlenecks of data flow, but also helps technical personnel ensure the accuracy and consistency of system development and operation. At the same time, managers can better understand complex data relationships through visual charts, thereby optimizing decision-making processes and improving overall operational efficiency.

[0120] Although operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order.Multitasking and parallel processing may be advantageous under certain circumstances.

[0121] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0122] The embodiment of the present invention further provides a data lineage relationship display system for the above method embodiment, including: a graphical display module, a search module and an interaction module.

[0123] The graphical display module is used to display the data lineage relationship of data entities in the form of charts.

[0124] The graphical display module is specifically used to visually present the blood relationship between data entities in the form of diagrams. First, determine the key data entities that need to be displayed. These entities include scenarios, job flows, jobs, data tables, views, and fields. On the big data cloud platform, they may present single-layer or multi-layer upstream and downstream blood relationships. Next, define a specific display form for each entity: the job flow is displayed through a flowchart composed of multiple rectangular boxes, and the job is represented by lines or rectangular boxes. Among them, a single solid line represents a general job, a single dotted line represents a view creation job, and a double solid line is used for synchronization jobs. Data tables and views are represented by circles, with solid circles representing data tables and hollow circles representing views. Through these graphical identification methods, data entities and their relationships can be clearly displayed.

[0125] The search module is used to search through the blood relationship dimension table according to the search conditions and return a result list.

[0126] The search module is specifically used to search through the lineage dimension table according to the search conditions set by the user, and return a list of results that meet the conditions. Users can set search conditions, such as tables, job flows, whether job nodes are running abnormally, etc., and use these parameters to locate upstream and downstream types. After the search is completed, the system displays the results in a list format and displays some attribute information at the same time. In addition, in the developed result display page, users can click on subjects such as job flows or tables, and the subject will serve as the center, expanding upward and downward to display three layers of lineage information.

[0127] The interactive module is used to support users to click on data entities and their connecting lines to expand the next level of data lineage relationships.

[0128] The interactive module is specifically used to support users to expand and view the data lineage relationship of the next level by clicking on the data entity and its connection line. In the lineage display diagram, each main node and the connection between nodes have interactive functions. For example, after the user clicks on the table node, they can view the attribute information of the table and switch to the structure display or partial data display of the table. On the structure page of the table details, after clicking on the field, the user can further expand and view the lineage relationship at the field level. In addition, by clicking on the job flow, the user can view the node details of the job flow and explore its upstream and downstream job flows, so as to continue to analyze its lineage relationship in depth.

[0129] The data lineage display system provided by the embodiment of the present invention intuitively presents the lineage of data entities through a graphical display module, combined with the precise retrieval capability provided by the search module and the hierarchical expansion function of the interactive module, so that users can easily explore and analyze complex data dependency structures, significantly improving the efficiency and depth of data management and analysis.

[0130] In order to facilitate understanding of the operation process of the data kinship display system provided by the embodiment of the present invention, Figures 2 to 9 For example, take the bank loan business scenario as an example. Figure 2 Searching for "loan application" in the scenario search box of the system list shown will display the related tables and job flows.

[0131] First, for business personnel, if they want to know what work has been done in the current loan business scenario, they can Figure 2 Click a job stream name in the system list to jump to Figure 8 The work flow page shown. Here, through the relationship map between work flows or between operations, business personnel can find the "upstream work flow", such as loan risk assessment, loan approval process, etc., and further click to view details. In this way, business personnel can fully understand the working situation of the loan business, clearly see the current approval strategy and key customer groups, and provide a basis for exploring new business growth points.

[0132] Secondly, for scientific and technical personnel, when judging whether the code of the "upstream operation flow" can be directly referenced, they can analyze it by tracing the table lineage and field lineage. Figure 2 Click a table in the system list to jump to Figure 3 The data lineage page shown. On this page, you can find the original tables and self-built intermediate tables related to the research loan promotion. After clicking to view these tables, you can enter the background code viewing page. Technology personnel can confirm the accuracy of the development work and whether it meets business needs, and achieve data sharing by using qualified intermediate tables to avoid duplicate development.

[0133] based on Figure 3If the user clicks on any solid line, the following data lineage page will be displayed: Figure 4 If the user clicks on any dotted line, Figure 5 If the user clicks on any circular pattern, the following Figure 6 If the user clicks on any double solid line, the following pop-up window will be displayed: Figure 6 The double solid line floating frame shown. Fig. 9 Click any field on the table structure page of the table details to expand the field-level lineage relationship; click a job flow to drill down to the node display of the job flow, and display its upstream and downstream job flows at the same time, so that you can continue to explore the upstream and downstream job flows.

[0134] Finally, for managers, Figure 2 The "Person in charge" and "Is it running normally" filter items in the system list shown pay attention to the number of completed jobs and abnormal situations of employees. Through the display method of table lineage, taking the self-built intermediate table t101_he_bims_warn_info as an example, a single solid line points to its source job. Click the dot on the single solid line to display the job name and view the details of the related data table. For the job lineage display, several job nodes in the current job flow are dependently associated through directed edges to help managers track and optimize the job process.

[0135] The data lineage display system provided by the embodiment of the present invention supports data tracking and management across multiple databases (including but not limited to bims, bims2, aca, ngb, oracle and other databases), provides comprehensive data management, quality assessment and monitoring functions, and significantly improves the transparency and efficiency of data operations. The system displays four levels of job flow lineage, job (node) lineage, table lineage and field lineage. Through the various forms of representation of graph structure and table structure, the information display is more comprehensive and specific. The system promotes data sharing and the creation of modular job flows, and reduces the generation of redundant job flows. In banking and financial business scenarios, existing job flows can be quickly integrated to meet new requirements. By screening and displaying detailed processing information, the system can quickly locate the root cause of data quality problems, support business analysis and risk avoidance, ensure the normal operation of data job flows, and provide strong support for business development.

[0136] Corresponding to the above method embodiment, the embodiment of the present invention also provides a data kinship relationship construction device, whose structure is as follows: Fig.10 As shown, it may include: a workflow description information obtaining unit, a scene category determining unit and a relationship parsing unit.

[0137] A job flow description information obtaining unit, used to obtain description information of a plurality of job flows, wherein the job flow includes a plurality of execution nodes, each execution node is provided with a node running path, and the description information includes codes and configuration data related to the job flow;

[0138] A scene category determination unit, used to determine the scene category to which each job flow belongs by using the description information;

[0139] The relationship parsing unit is used to parse the relationships between job flows, nodes, and tables for each job flow under any scenario category, and to build relationships between job flows, job flow and node, node and table, table and view, and field through dependency events, node operation path diagram, and SQL statement analysis.

[0140] Optionally, the relationship parsing unit can be specifically used to construct the first layer of data lineage relationship between each job flow based on the dependency events in the configuration data of each job flow; construct the second layer of data lineage relationship between each job flow and each execution node it contains; for the same job flow: construct the third layer of data lineage relationship between each execution node based on the node running path of each execution node contained in the job flow; perform logical judgment based on the code content of each execution node to confirm whether there is a node table for each execution node; for execution nodes with node tables: construct the fourth layer of data lineage relationship between the execution node and its node table; construct the fifth layer of data lineage relationship between each node table based on the prefix code of each node table in the corresponding code content; construct the sixth layer of data lineage relationship between the node table and the view based on the view construction statement of each node table in the corresponding code content; for any node table: construct the seventh layer of data lineage relationship between the fields in the node table based on the execution statement of the node table in the corresponding code content.

[0141] Optionally, the job flow related code and configuration data include basic information of the job flow, input and output dependencies, historical operation data, database table information and properties.

[0142] Optionally, the data blood relationship construction device may further include: a data table construction unit.

[0143] The data table construction unit is used to construct the lineage dimension table and other data tables storing subject attribute information according to the relationships between job flows, job flows and nodes, nodes, nodes and tables, tables, tables and views, and fields after the relationship analysis unit constructs the relationships between job flows, job flows and nodes, nodes, nodes and tables, tables, tables and views, and fields.

[0144] Optionally, other data tables storing subject attribute information include at least one of a job flow information table, a table / view information table, a job node information table, and a field information table.

[0145] Optionally, the data lineage relationship construction device may further include: a target data entity obtaining unit and a data lineage relationship display unit.

[0146] The target data entity obtaining unit is used to obtain a target data entity to be displayed, wherein the data entity includes at least one entity of a scene, a job flow, a job, a data table, a view and a field.

[0147] The data lineage relationship display unit is used to display the data lineage relationship between the target data entity and other data entities in the form of a chart based on the lineage dimension table related to the target data entity.

[0148] The data lineage relationship construction device provided by the present invention is used to: obtain description information of multiple job flows, wherein the job flow includes multiple execution nodes, each execution node is provided with a node running path, and the description information includes code and configuration data related to the job flow; using the description information, determine the scene category to which each job flow belongs; for each job flow under any scene category: parse the relationship between the job flow, nodes, and tables, and construct the relationship between job flows, the relationship between job flows and nodes, the relationship between nodes, the relationship between nodes and tables, the relationship between tables, the relationship between tables and views, and the relationship between fields through dependency events, node running path diagrams, and SQL statement analysis. The present invention can comprehensively construct multiple relationships between data hierarchies by parsing and analyzing the description information of the job flow and its nodes, thereby enhancing the accuracy and depth of data lineage relationship analysis.

[0149] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0150] The data blood relationship construction device includes a processor and a memory. The above-mentioned work flow description information acquisition unit, scene category determination unit and relationship analysis unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0151] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the description information of the job flow and its nodes can be parsed and analyzed by adjusting the kernel parameters, and multiple relationships between data levels can be fully constructed, thereby enhancing the accuracy and depth of data lineage relationship analysis.

[0152] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the data blood relationship construction method is implemented.

[0153] An embodiment of the present invention provides a processor, which is used to run a program, wherein the data blood relationship construction method is executed when the program is running.

[0154] like Fig.11 As shown, an embodiment of the present invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003; wherein the processor 1001 and the memory 1002 communicate with each other through the bus 1003; the processor 1001 is used to call the program instructions in the memory 1002 to execute the above-mentioned data lineage relationship construction method. The electronic device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0155] The present invention also provides a computer program product, which, when executed on an electronic device, is suitable for executing a program that initializes the steps of the data kinship relationship construction method.

[0156] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] In a typical configuration, an electronic device includes one or more processors (CPU), a memory, and a bus. The electronic device may also include an input / output interface, a network interface, and the like.

[0158] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0159] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0160] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0161] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0162] In the description of the present invention, it should be understood that the terms "up", "down", "front", "back", "left" and "right" etc. indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the positions or elements referred to must have specific directions, be constructed and operate in specific directions. Therefore, they should not be understood as limitations of the present invention.

[0163] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0164] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0165] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A method for constructing data kinship relationships, characterized in that: include: Obtaining description information of a plurality of job flows, wherein the job flows include a plurality of execution nodes, each of the execution nodes is provided with a node running path, and the description information includes code and configuration data related to the job flows; Using the description information, determining the scene category to which each of the job flows belongs; For each of the job flows under any scenario category: parse the relationship between job flows, nodes, and tables, and build the relationship between job flows, the relationship between job flows and nodes, the relationship between nodes, the relationship between nodes and tables, the relationship between tables, the relationship between tables and views, and the relationship between fields through dependency events, node operation path diagrams, and SQL statement analysis.

2. The method according to claim 1, characterized in that The above-mentioned process constructs the relationship between job flows, the relationship between job flows and nodes, the relationship between nodes, the relationship between nodes and tables, the relationship between tables, the relationship between tables and views, and the relationship between fields through the analysis of dependency events, node operation path diagrams, and SQL statements, including: According to the dependency events in the configuration data of each of the operation flows, a first-level data lineage relationship between the operation flows is established; respectively constructing the second-layer data kinship relationship between each of the operation flows and each of the execution nodes contained therein; For the same job flow: constructing a third-layer data kinship relationship between the execution nodes according to the node running paths of the execution nodes included in the job flow; Performing logical judgment according to the code content of each execution node to confirm whether a node table exists for each execution node; For the execution node having the node table: constructing a fourth-layer data lineage relationship between the execution node and its node table; According to the preceding codes of the node tables in the corresponding code content, a fifth layer of data kinship relationship between the node tables is constructed; According to the view construction statements of each node table in the corresponding code content, a sixth layer of data kinship relationship between the node table and the view is constructed; For any of the node tables: according to the execution statements of the node table in the corresponding code content, the seventh layer of data kinship relationship between the fields in the node table is constructed.

3. The method according to claim 1, characterized in that The code and configuration data related to the job flow include basic information, input and output dependencies, historical operation data, database table information and attributes of the job flow.

4. The method according to claim 1, characterized in that After constructing the relationship between job flows, the relationship between job flows and nodes, the relationship between nodes, the relationship between nodes and tables, the relationship between tables, the relationship between tables and views, and the relationship between fields, the method further includes: According to the relationship between the job flows, the relationship between the job flows and the nodes, the relationship between the nodes, the relationship between the nodes and the tables, the relationship between the tables, the relationship between the tables and the views, and the relationship between the fields, a lineage dimension table and other data tables storing subject attribute information are constructed.

5. The method according to claim 4, characterized in that The other data tables storing subject attribute information include at least one data table among a job flow information table, a table / view information table, a job node information table and a field information table.

6. The method according to any one of claims 4 to 5, characterized in that Also includes: Obtaining a target data entity to be displayed, wherein the data entity includes at least one entity of a scene, a job flow, a job, a data table, a view, and a field; Based on the lineage dimension table related to the target data entity, the data lineage relationship between the target data entity and other data entities is displayed in a graphical form.

7. A data kinship relationship construction device, characterized in that: include: The workflow description information acquisition unit, the scene category determination unit and the relationship analysis unit, The job flow description information obtaining unit is used to obtain description information of a plurality of job flows, wherein the job flow includes a plurality of execution nodes, each of the execution nodes is provided with a node running path, and the description information includes code and configuration data related to the job flow; The scene category determination unit is used to determine the scene category to which each of the job flows belongs by using the description information; The relationship parsing unit is used to parse the relationships among job flows, nodes, and tables for each job flow under any scenario category, and to construct relationships among job flows, job flow and node, node and table, table and view, and field through dependency events, node operation path diagram, and SQL statement analysis.

8. A data lineage relationship display system, characterized in that: include: Graphical display module, search module and interactive module, The graphical display module is used to display the data kinship relationship of data entities in the form of diagrams; The search module is used to search through the blood relationship dimension table according to the search conditions and return a result list; The interaction module is used to support the user to click on the data entity and its connection line to expand the data lineage relationship of the next level.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the data blood relationship construction method according to any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: The electronic device includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the data blood relationship construction method as described in any one of claims 1 to 6.

Images