Field-level bloodline resolution method and device, computer device, and storage medium

Abstract

The application relates to a field-level blood relationship analysis method and device, computer equipment and a storage medium. The method comprises the following steps: in the case that a field to be queried is acquired, a target mapping relationship matched with a query type is determined from a field mapping table according to the query type corresponding to the field to be queried; at least one target field is determined from the field mapping table according to the target mapping relationship; a full-link field dependency graph is generated according to the target field and the mapping relationship between the target fields; a field to be queried node corresponding to the field to be queried is determined from the full-link field dependency graph according to the field to be queried, and a field to be queried link is acquired from the full-link field dependency graph according to the field to be queried node; and a blood relationship analysis result of the field to be queried is obtained based on the field to be queried link. The method can prevent information from being missed in the field-level blood relationship analysis process, and improves the accuracy of the field-level blood relationship analysis.

Description

Technical Field

[0001] This application relates to the field of data warehouse technology, and in particular to a field-level lineage resolution method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] Current lineage analysis techniques typically focus on the entity lineage relationships at the table level and between fields, only informing users of the relationship between A and B, without specifying the processing methods and logic. For example, in the conditional statement CASE C WHEN D CASE E WHEN F, C and E are field constraints. Since these constrained fields lack underlying data transfer logic, conventional lineage analysis solutions do not consider the lineage relationship at this point. However, the values ​​at C and E directly affect the final return value of the query field. Traditional lineage analysis solutions lack effective means to analyze the intricate processing logic between fields, making it difficult to determine the true downstream data path. In real-world business scenarios, users often fail to analyze the constraints at the WHEN position, leading to inaccurate analysis and assessment, which in turn hinders business operations.

[0003] Current kinship analysis methods suffer from missing information, leading to inaccurate analysis and evaluation. Summary of the Invention

[0004] Therefore, it is necessary to provide a field-level lineage resolution method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of analysis and evaluation, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a field-level lineage resolution method. The method includes:

[0006] If the field to be queried is obtained, the target mapping relationship matching the query type is determined from the field mapping table according to the query type corresponding to the field to be queried. The field mapping table includes multiple fields and the mapping relationship between each field. The mapping relationship is used to represent the field processing information between upstream fields and downstream fields.

[0007] Based on the target mapping relationship, determine at least one target field from the field mapping table;

[0008] A full-link field dependency graph is generated based on the target fields and the mapping relationships between them. The full-link field dependency graph includes field nodes that correspond one-to-one with the target fields, as well as relationship lines between the field nodes to represent the mapping relationships.

[0009] Based on the field to be queried, the corresponding field node is determined from the full-link field dependency graph. Based on the field node, the field link is obtained from the full-link field dependency graph. Based on the field link, the lineage analysis result of the field to be queried is obtained.

[0010] In one embodiment, the query type is divided into a first query type and a second query type. Based on the query type corresponding to the field to be queried, a target mapping relationship matching the query type is determined from the field mapping table of the target data warehouse platform, including:

[0011] If the query type corresponding to the field to be queried is the first query type, the mapping relationship of the first mapping type in the field mapping table is used as the target mapping relationship;

[0012] If the query type corresponding to the field to be queried is the second query type, the mapping relationship of the first mapping type and the mapping relationship of the second mapping type in the field mapping table are used as the target mapping relationship;

[0013] The first mapping type is used to represent the mapping relationship between upstream and downstream fields without any restrictions; the second mapping type is used to represent the mapping relationship between upstream and downstream fields with restrictions.

[0014] In one embodiment, determining the target field from the field mapping table based on the target mapping relationship includes:

[0015] From the field mapping table, determine the upstream and downstream fields corresponding to the target mapping relationship, and use the upstream and downstream fields corresponding to the target mapping relationship as the target fields.

[0016] In one embodiment, a full-link field dependency graph is generated based on the target fields and the mapping relationships between the target fields, including:

[0017] Based on the mapping relationship between each target field, determine the upstream and downstream relationships between each target field;

[0018] Based on the upstream and downstream relationships between each target field, determine at least one starting node of the full-link field dependency graph;

[0019] Based on the upstream and downstream relationships between each target field, starting from at least one starting node, field nodes corresponding one-to-one with each target field are generated, and relationship connections matching the corresponding mapping relationships are generated between each field node to obtain a full-link field dependency graph.

[0020] In one embodiment, determining at least one starting node of the end-to-end field dependency graph based on the upstream and downstream relationships between the fields includes:

[0021] If a field does not have a corresponding upstream field, but has a corresponding downstream field, then the field will be used as the starting field.

[0022] Generate a starting node based on the starting field.

[0023] In one embodiment, generating relationship connections between field nodes that match the corresponding mapping relationships includes:

[0024] If the mapping relationship between two field nodes is of the first mapping type, a relationship connection of the first representation form is generated between the two fields; the first representation form is used to indicate that there are no restrictions on the mapping relationship between the two field nodes.

[0025] If the mapping relationship between two field nodes is of the second mapping type, a relationship connection with a second representation is generated between the two fields; the second representation is used to characterize that there are restrictions on the mapping relationship between the two field nodes.

[0026] The first representation of the relationship connection and the second representation of the relationship connection are linked by the upstream field node pointing to the downstream field node in the two field nodes.

[0027] In one embodiment, the query field link is obtained from the full-link field dependency graph based on the query field node, including:

[0028] Based on the query path corresponding to the field to be queried, determine the relevant field nodes corresponding to the node of the field to be queried from the full-link field dependency graph;

[0029] Based on the field node to be queried and related field nodes, obtain the link of the field to be queried from the full-link field dependency graph.

[0030] In one embodiment, the query path is divided into upstream link query and downstream link query. Based on the query path corresponding to the field to be queried, the relevant field nodes corresponding to the node of the field to be queried are determined from the full-link field dependency graph, including:

[0031] Based on the mapping relationship between the field nodes in the full-link field dependency graph, determine the upstream and downstream field nodes corresponding to the field to be queried;

[0032] If the query path corresponding to the field to be queried is an upstream link query, the upstream field node corresponding to the field to be queried is obtained from the full link field dependency graph as the relevant field node;

[0033] If the query path corresponding to the field to be queried is a downstream link query, the downstream field node of the field to be queried is obtained from the full link field dependency graph as the relevant field node.

[0034] In one embodiment, if a field to be queried is obtained, before determining the target mapping relationship matching the query type from the field mapping table based on the query type corresponding to the field to be queried, the method further includes:

[0035] Retrieve query information; the query information includes the field to be queried, as well as the query type and query path corresponding to the field to be queried.

[0036] In one embodiment, the method further includes:

[0037] Construct an abstract syntax tree based on script files from the target data warehouse platform;

[0038] Obtain the mapping relationship between adjacent upstream and downstream fields based on the abstract syntax tree, and construct a field mapping table based on the mapping relationship.

[0039] Secondly, this application also provides a field-level lineage analysis device. The device includes:

[0040] The acquisition module, if a field to be queried is acquired, determines the target mapping relationship matching the query type from the field mapping table based on the query type corresponding to the field to be queried. The field mapping table includes multiple fields and the mapping relationships between each field. The mapping relationship is used to represent the field processing information between upstream and downstream fields.

[0041] The query module is used to determine at least one target field from the field mapping table based on the target mapping relationship;

[0042] The generation module is used to generate a full-link field dependency graph based on the target fields and the mapping relationships between them. The full-link field dependency graph includes field nodes that correspond one-to-one with the target fields, as well as relationship lines between the field nodes to represent the mapping relationships.

[0043] The filtering module is used to determine the corresponding query field node from the full-link field dependency graph based on the query field, obtain the query field link from the full-link field dependency graph based on the query field node, and obtain the lineage analysis result of the query field based on the query field link.

[0044] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0045] If the field to be queried is obtained, the target mapping relationship matching the query type is determined from the field mapping table according to the query type corresponding to the field to be queried. The field mapping table includes multiple fields and the mapping relationship between each field. The mapping relationship is used to represent the field processing information between upstream fields and downstream fields.

[0046] Based on the target mapping relationship, determine at least one target field from the field mapping table;

[0047] A full-link field dependency graph is generated based on the target fields and the mapping relationships between them. The full-link field dependency graph includes field nodes that correspond one-to-one with the target fields, as well as relationship lines between the field nodes to represent the mapping relationships.

[0048] Based on the field to be queried, the corresponding field node is determined from the full-link field dependency graph. Based on the field node, the field link is obtained from the full-link field dependency graph. Based on the field link, the lineage analysis result of the field to be queried is obtained.

[0049] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0050] If the field to be queried is obtained, the target mapping relationship matching the query type is determined from the field mapping table according to the query type corresponding to the field to be queried. The field mapping table includes multiple fields and the mapping relationship between each field. The mapping relationship is used to represent the field processing information between upstream fields and downstream fields.

[0051] Based on the target mapping relationship, determine at least one target field from the field mapping table;

[0052] A full-link field dependency graph is generated based on the target fields and the mapping relationships between them. The full-link field dependency graph includes field nodes that correspond one-to-one with the target fields, as well as relationship lines between the field nodes to represent the mapping relationships.

[0053] Based on the field to be queried, the corresponding field node is determined from the full-link field dependency graph. Based on the field node, the field link is obtained from the full-link field dependency graph. Based on the field link, the lineage analysis result of the field to be queried is obtained.

[0054] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0055] If the field to be queried is obtained, the target mapping relationship matching the query type is determined from the field mapping table according to the query type corresponding to the field to be queried. The field mapping table includes multiple fields and the mapping relationship between each field. The mapping relationship is used to represent the field processing information between upstream fields and downstream fields.

[0056] Based on the target mapping relationship, determine at least one target field from the field mapping table;

[0057] A full-link field dependency graph is generated based on the target fields and the mapping relationships between them. The full-link field dependency graph includes field nodes that correspond one-to-one with the target fields, as well as relationship lines between the field nodes to represent the mapping relationships.

[0058] Based on the field to be queried, the corresponding field node is determined from the full-link field dependency graph. Based on the field node, the field link is obtained from the full-link field dependency graph. Based on the field link, the lineage analysis result of the field to be queried is obtained.

[0059] The aforementioned field-level lineage analysis method, apparatus, computer equipment, storage medium, and computer program product, upon obtaining the field to be queried, determine the target mapping relationship matching the query type from the field mapping table based on the query type corresponding to the field to be queried. Then, based on the target mapping relationship, at least one target field is determined from the field mapping table. Finally, a full-link field dependency graph is generated based on the target fields and the mapping relationships between each target field. The full-link field dependency graph is only generated when a query request is received, and field data can be selected from the field mapping table as needed, thereby saving equipment resources. Furthermore, based on the field to be queried, the corresponding query field node is determined from the full-link field dependency graph, and based on the query field node, the query field link is obtained from the full-link field dependency graph. Based on the query field link, the lineage analysis result of the query field is obtained. In this way, based on the query field node, the query field link related to the query field can be obtained from the full-link field dependency graph, and based on the query field link, the lineage analysis result of the query field can be obtained. This prevents information omissions during field-level lineage analysis and improves the accuracy of field-level lineage analysis. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating a field-level lineage resolution method in one embodiment;

[0061] Figure 2 This is a schematic diagram of a full-link field dependency graph that includes only the first mapping category in one embodiment;

[0062] Figure 3 This is a schematic diagram of a full-link field dependency graph including a first mapping category and a second mapping category in one embodiment;

[0063] Figure 4 This is a flowchart illustrating the logic of a field-level lineage resolution method in one embodiment.

[0064] Figure 5 This is a structural block diagram of a field-level lineage resolution device in one embodiment;

[0065] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0067] In one embodiment, a field-level lineage resolution method is provided. This embodiment illustrates the method by applying it to a computer device, which can be a terminal or a server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart medical devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. The server can be a standalone server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:

[0068] Step 102: If the field to be queried is obtained, then the target mapping relationship matching the query type is determined from the field mapping table according to the query type corresponding to the field to be queried. The field mapping table includes multiple fields and the mapping relationship between each field. The mapping relationship is used to represent the field processing information between upstream fields and downstream fields.

[0069] The field mapping table is generated based on the data information in the Abstract Syntax Tree (AST), which is constructed based on the ETL (Extract-Transform-Load, data warehouse technology) platform and related files in the database. The field mapping table can be stored directly on a computer device, on a server capable of communicating with the computer device, or on an external storage device that can connect to the computer device and perform data transfer.

[0070] Optionally, the computer device receives query information, identifies the field to be queried and the query type from the query information, and filters the target mapping relationships that match the query type in the field mapping table according to the query type. The field mapping table typically includes multiple fields, source information for each field, and mapping relationships (field processing information) between every two fields. A field may have mapping relationships with multiple other fields. A field can be an upstream field of another field and a downstream field of a third field. Generally, between two fields with a mapping relationship, the upstream field refers to the source field before field processing, and the downstream field refers to the target field after field processing. A field can appear in both the source and target fields simultaneously. For example, if field A is processed by method XX to obtain field B, and field B is processed by method YY to obtain field C, then field A is the upstream field of field B, and field B is the upstream field of field A. Alternatively, field A can be said to be the source field of field B, and field B is the target field of field A. The mapping relationship between field A and field B is the XX processing method. Similarly, if field B is the upstream field of field C, and field C is the upstream field of field B, then field B can be said to be the source field of field C, and field C is the target field of field B. The mapping relationship between field B and field C is the YY processing method.

[0071] In a feasible implementation, the field mapping table also includes source information for each field. Source information includes table-level and library-level sources. For example, in a field mapping table like Table 1, the first three data items (source library, source table, source field) represent the constituent objects of the source end, the last three data items (target library, target table, target field) represent the constituent objects of the target end, and the last target data expression is the logical data composition of the target field, i.e., the specific mapping logic from source data to target data. The first six data items only focus on the relational dependencies between upstream and downstream table fields (equivalent to whether a mapping relationship exists), and the last item describes the specific form of these relational dependencies (equivalent to the specific mapping relationship between two fields). Traditional lineage analysis schemes typically only focus on the dependency relationships between upstream and downstream table fields. This patent application proposes to refine the data lineage granularity to the field level while providing complete processing logic for any field during the transformation process, including the source table name and source field, and incorporating this processing logic into the lineage link impact analysis algorithm. This enables the location and tracing of data lineage issues and solves the problem of inaccurate downstream impact analysis.

[0072]

[0073] Table 1

[0074] Step 104: Determine at least one target field from the field mapping table based on the target mapping relationship.

[0075] Optionally, the computer device, based on the determined target mapping relationship, traverses the upstream and downstream fields corresponding to each target mapping relationship in the field mapping table one by one to obtain all target fields.

[0076] Step 106: Generate a full-link field dependency graph based on the target fields and the mapping relationships between them. The full-link field dependency graph includes field nodes that correspond one-to-one with the target fields, as well as relationship lines between the field nodes to represent the mapping relationships.

[0077] Optionally, the computer device generates field nodes based on the target fields, and generates connections between field nodes based on the mapping relationship between every two target fields, thus generating a full-link field dependency graph that includes all target fields and the mapping relationships between each target field.

[0078] Step 108: Based on the field to be queried, determine the node corresponding to the field to be queried from the full-link field dependency graph, and obtain the link of the field to be queried from the full-link field dependency graph based on the node of the field to be queried, and obtain the lineage analysis result of the field to be queried based on the link of the field to be queried.

[0079] Optionally, the computer device, based on the query field identified from the query information, determines the corresponding query field node from a temporarily generated end-to-end field dependency graph. Then, it extracts the query field link containing the query field node. This query field link can be a path starting from the query field node and extending downstream to the end node of the end-to-end field dependency graph; it can also be a path starting from the query field node and extending upstream to the beginning node of the end-to-end field dependency graph; or it can be a path including the query field node from the beginning node to the end node of the end-to-end field dependency graph. Furthermore, the computer device generates a lineage analysis result for the query field based on the query field link, or it can directly use the query field link as the lineage analysis result for the query field.

[0080] In one feasible implementation, after the computer device obtains the query field link from the full-link field dependency graph, it deletes the full-link field dependency graph and retains only the query field link, thereby saving device resources and reducing device load.

[0081] In the above field-level lineage analysis method, upon obtaining the field to be queried, a target mapping relationship matching the query type is determined from the field mapping table based on the query type corresponding to the field. Then, based on the target mapping relationship, at least one target field is determined from the field mapping table. Finally, a full-link field dependency graph is generated based on the target fields and the mapping relationships between them. The full-link field dependency graph is only generated when a query request is received, and field data can be selected from the field mapping table as needed, thus saving device resources. Furthermore, based on the field to be queried, the corresponding field node is determined from the full-link field dependency graph, and the field link is obtained from the full-link field dependency graph based on the field node. The lineage analysis result of the field to be queried is then obtained based on the field link. In this way, the related field links can be obtained from the full-link field dependency graph based on the field node, and the lineage analysis result of the field to be queried can be obtained based on the field link. This prevents information omissions during field-level lineage analysis and improves the accuracy of field-level lineage analysis.

[0082] In one embodiment, the query type is divided into a first query type and a second query type. Based on the query type corresponding to the field to be queried, a target mapping relationship matching the query type is determined from the field mapping table of the target data warehouse platform. This includes: if the query type corresponding to the field to be queried is the first query type, the mapping relationship of the first mapping type in the field mapping table is used as the target mapping relationship; if the query type corresponding to the field to be queried is the second query type, the mapping relationship of the first mapping type and the mapping relationship of the second mapping type in the field mapping table are used as the target mapping relationship. The first mapping type is used to indicate that there are no restrictions on the mapping relationship between upstream and downstream fields; the second mapping type is used to indicate that there are restrictions on the mapping relationship between upstream and downstream fields.

[0083] The mapping relationship has restrictions, meaning that the upstream field will be processed to obtain the downstream field only under certain conditions.

[0084] Optionally, in the field mapping table, each mapping relationship is pre-classified into mapping types. Mapping types with restrictions are labeled with a second mapping type, while mapping relationships without restrictions are labeled with a first mapping type. A correspondence between mapping types and query types is established: the first query type corresponds to the first mapping type, and the second query type corresponds to both the first and second mapping types. This is equivalent to pre-configuring two query types, with the second query type providing more detailed information than the first query type. When the computer device receives query information and identifies the query type corresponding to the field to be queried, it determines the label of the mapping type to be searched based on the correspondence between mapping types and query types. If the query type is the first query type, the mapping relationship of the first mapping type in the field mapping table is used as the target mapping relationship; if the query type corresponding to the field to be queried is the second query type, the mapping relationship of the first and second mapping types in the field mapping table is used as the target mapping relationship.

[0085] In this embodiment, if the query type corresponding to the field to be queried is a first query type, the mapping relationship of the first mapping type in the field mapping table is used as the target mapping relationship; if the query type corresponding to the field to be queried is a second query type, the mapping relationship of the first mapping type and the mapping relationship of the second mapping type in the field mapping table are used as the target mapping relationship. The first mapping type is used to indicate that there are no restrictions on the mapping relationship between the upstream and downstream fields; the second mapping type is used to indicate that there are restrictions on the mapping relationship between the upstream and downstream fields. Different quantities and types of field-related data can be selected from the field mapping table according to different query requirements, which can improve the operating speed of the device and thus improve the efficiency of lineage resolution.

[0086] In one embodiment, determining the target field from the field mapping table according to the target mapping relationship includes: determining the upstream field and downstream field corresponding to the target mapping relationship from the field mapping table, and using the upstream field and downstream field corresponding to the target mapping relationship as the target field.

[0087] Optionally, as shown in Table 1, after determining the target mapping relationship, the computer device takes the source field (upstream field) and target field (downstream field) in the same row of a target mapping relationship as two target fields, and iterates through the source field and target field in the same row of each target mapping relationship to obtain all the target fields in this query process.

[0088] In this embodiment, the upstream and downstream fields corresponding to the target mapping relationship are determined from the field mapping table, and these upstream and downstream fields are used as the target fields. This allows for the automatic determination of all target fields from the field mapping table based on the selected target mapping relationship, thus improving the efficiency of lineage resolution.

[0089] In one embodiment, generating a full-link field dependency graph based on target fields and the mapping relationships between them includes: determining the upstream and downstream relationships between target fields based on the mapping relationships; if a field does not have a corresponding upstream field but has a corresponding downstream field, the field is used as a starting field; generating a starting node based on the starting field. Based on the upstream and downstream relationships between target fields, starting from at least one starting node, field nodes corresponding one-to-one with each target field are generated. If the mapping relationship between two field nodes is of a first mapping type, a relationship connection of a first representation is generated between the two fields; the first representation is used to characterize that the mapping relationship between two field nodes has no restrictions; if the mapping relationship between two field nodes is of a second mapping type, a relationship connection of a second representation is generated between the two fields; thus obtaining the full-link field dependency graph. The second representation is used to characterize that the mapping relationship between two field nodes has restrictions; the relationship connection of the first representation and the relationship connection of the second representation point from the upstream field node to the downstream field node in the two field nodes.

[0090] Among them, the relationship connection in the first form can be represented by a solid line, and the relationship connection in the second form can be represented by a dashed line.

[0091] Optionally, the field mapping table shown in Table 1 is traversed, and the set difference between the source field set SET_SRC (source library, source table, source field) and the target field set SET_TAR (target library, target table, target field) is calculated. The source field objects (set differences) that do not exist in the target field set are the starting nodes of the global dependency graph. This is equivalent to finding field nodes that do not have corresponding upstream nodes but only corresponding downstream nodes as starting nodes. The starting nodes are stored in the `starts` array structure, and subsequent path generation can be performed for each starting node. Here, the specific content of the starting node is defined as either {library, table, field} or {library, table}. For example, {SO.CPG.BUMENN} or {SO.CPG} can be considered a graph node.

[0092] Upon receiving the first query type, after the field mapping table for the view and batch job is generated, the starting node extracted above or any other custom node is used as the starting point of the link. A node dependency graph G describing the mapping relationships between the field nodes is generated using the depth-first search (DFS) algorithm. The main process is as follows:

[0093] Collect the desired link starting points according to user needs and set them as node set S; preset a node access status set P, which by default contains all nodes in the field mapping table and all nodes are in an unaccessed state; preset a node stack C to be accessed, which is empty by default. In addition, a special filter node set F can be selected according to user needs (for example, when querying the blood relationship of a specified interval), which can control the combination logic of link nodes during link generation.

[0094] Nodes from the user-specified starting set S are pushed onto the stack C of nodes to be visited in sequence. The stack itself is a set, but its access logic is fixed as Last-In-First-Out (LIFO).

[0095] Pop a node N0 from the stack of nodes to be visited C, use node N0 as the source in the field mapping table in step 4, and extract the corresponding target object Nx (there may be multiple target nodes) from the field mapping table.

[0096] If the target object Nx is empty, then skip directly to process 6; otherwise, at this time, it may be possible to check whether these objects meet the specific link generation logic, such as whether these nodes are included in the special filter node set F. Unvisited nodes that do not meet the filtering requirements and are not currently in the node stack C to be visited will be inserted into the node stack C to be visited, and at the same time, the access status of node N0 in the state set P will be set to visited.

[0097] The graph is stored using key-value pairs (K / V), where K represents the source node and V represents the destination node, and deduplication is performed. Node N0 from process 3 is used as K, and Nx as V; these key-value pairs are then inserted into graph G.

[0098] Repeat steps 3, 4, and 5: continuously pop a node from the stack of nodes to be visited (C) and visit it, until the stack of nodes to be visited (C) is empty, thus generating the final node dependency graph G, as shown below. Figure 2 As shown, it can be seen that Figure 2 It contains only field nodes whose mapping relationship is the first mapping type (solid line).

[0099] In the node dependency graph G, nodes can be table-level or field-level, and there are multiple different types of nodes, such as permanent tables T (see T1, T2, etc. in the diagram), temporary tables TT (see TT1, TT2 in the diagram), and views V (see V1, V2, etc. in the diagram). A path of length 1 (such as T2-V2) may represent a relational dependency existing in the view logic, that is, information from the entity table is transferred to the view, while a path of length greater than 1 (such as T1-TT2-T5) may represent a relational dependency existing in the ETL job logic, that is, information is transferred sequentially between different nodes.

[0100] Furthermore, since table-level dependencies are derived from field-level dependencies, the field-level node dependency graph essentially already contains the table-level dependency graph.

[0101] In this context, Cx represents field-level nodes, which consist of {database, table, field}. A is a marker node, and all upstream field nodes pointing to this node are flow restriction nodes. For example, in the diagram above, the two adjacent nodes (T2, A) indicate that in a certain SQL statement, node T2 was used as a data retrieval restriction field to filter out data that was not needed by the target.

[0102] When a computer device receives a second query type, and when analyzing field-level lineage relationships, based on... Figure 2 For each mapping relationship of the second mapping type in the generated node dependency graph G, an auxiliary restricted flow is constructed, such as... Figure 3 As shown. Typically, in the logic of views or ETL jobs, there are often restrictions on certain fields during the data processing stage to filter target data. For example, in the conditional statement `CASE C WHEN CASE E WHEN F`, C and E are field restrictions. Since these restricted fields do not have underlying data transfer logic in practice, they are not reflected in the data processing logic. Figure 2 In the data stream, however, the values ​​at C and E will directly affect the return value of the final target field, therefore... Figure 2 Further processing was performed to obtain... Figure 3 As you can see, Figure 3 It contains field nodes with mapping relationships of the first mapping type (solid line) and the second mapping type (dashed line).

[0103] In this context, Cx represents field-level nodes, which consist of {database, table, field}. A is a marker node, and all upstream field nodes pointing to this node are flow restriction nodes. For example, in the diagram above, the two adjacent nodes (T2, A) indicate that in a certain SQL statement, node T2 was used as a data retrieval restriction field to filter out data that was not needed by the target.

[0104] In this embodiment, the upstream and downstream relationships between target fields are determined based on the mapping relationships between them. Based on these relationships, at least one starting node of the full-link field dependency graph is determined. Starting from this node, field nodes corresponding one-to-one with each target field are generated, and connection lines matching the corresponding mapping relationships are created between these field nodes, thus obtaining the full-link field dependency graph. This method can generate full-link field dependency graphs containing different amounts of information according to different query requirements, improving both the accuracy and efficiency of field-level lineage resolution.

[0105] In one embodiment, obtaining the query field link from the full-link field dependency graph based on the query field node includes: dividing the query path into upstream link query and downstream link query; determining the upstream and downstream field nodes corresponding to the query field based on the mapping relationship between each field node in the full-link field dependency graph; if the query path corresponding to the query field is an upstream link query, obtaining the upstream field node corresponding to the query field as the relevant field node from the full-link field dependency graph; if the query path corresponding to the query field is a downstream link query, obtaining the downstream field node of the query field node as the relevant field node from the full-link field dependency graph. The query field link is then obtained from the full-link field dependency graph based on the query field node and the relevant field nodes.

[0106] Optionally, the query information may also include query path information. The computer device identifies the query path from the query information. The query path can be divided into upstream link query and downstream link query. If the query path corresponding to the field to be queried is an upstream link query, the computer device obtains the upstream field node corresponding to the field to be queried as the relevant field node from the full link field dependency graph. The query field link is the path that starts from the field node to be queried, extends downstream of the field node to the relevant field node, and continues until the end node of the full link field dependency graph. If the query path corresponding to the field to be queried is a downstream link query, the computer device obtains the downstream field node of the field to be queried as the relevant field node from the full link field dependency graph. The query field link is the path that starts from the field node to be queried, extends upstream of the field node to the relevant field node, and continues until the start node of the full link field dependency graph.

[0107] In this embodiment, based on the query path corresponding to the field to be queried, the relevant field nodes corresponding to the node of the field to be queried are determined from the full-link field dependency graph; based on the node of the field to be queried and the relevant field nodes, the link of the field to be queried is obtained from the full-link field dependency graph. This allows for the generation of links of the field to be queried containing different amounts of information according to different query requirements, improving both the accuracy and efficiency of field-level lineage analysis.

[0108] In one embodiment, if a field to be queried is obtained, before determining the target mapping relationship matching the query type from the field mapping table based on the query type corresponding to the field to be queried, the method further includes: obtaining query information; the query information includes the field to be queried, as well as the query type and query path corresponding to the field to be queried.

[0109] Optionally, when the computer device receives query information, it identifies the query information. Only when the field to be queried in the query information is detected will a full-link field dependency graph be generated according to the query type, and the link of the field to be queried be selected from the full-link field dependency graph according to the query path.

[0110] In this embodiment, query information is obtained; the query information includes the field to be queried, as well as the query type and query path corresponding to the field to be queried. This can reduce the device load and improve the efficiency of device resource utilization when the lineage analysis task is not performed.

[0111] In one embodiment, the method further includes: constructing an abstract syntax tree based on a script file of the target data warehouse platform; obtaining the mapping relationship between adjacent upstream fields and downstream fields according to the abstract syntax tree; and constructing a field mapping table according to the mapping relationship.

[0112] Among them, the Abstract Syntax Tree (AST) is an abstract representation of the syntax structure, which is presented in a tree-like form. Each node in the tree represents a structure in the source code.

[0113] Optionally, the computer device pre-builds an abstract syntax tree based on the script file of the target data warehouse platform; obtains the mapping relationship between adjacent upstream and downstream fields according to the abstract syntax tree; constructs a field mapping table according to the mapping relationship; and stores the field mapping table to facilitate the execution of lineage resolution tasks.

[0114] Specifically, collect all ETL scripts. Considering that the purpose of SQL scripts is to output SQL to the data warehouse and execute it, the scripts are modified as follows to output complete SQL logic:

[0115] 1) At the beginning of the script, standard output is turned off, that is, standard output is redirected to / dev / null, and the script will not output any information thereafter.

[0116] 2) Locate the SQL output statement segment.

[0117] 3) Restore standard output before the SQL output statement segment.

[0118] 4) Continue to disable standard output after the SQL output statement segment.

[0119] The modified ETL script is executed to obtain the complete business logic for each batch job on that day based on the input data date parameter. This SQL logic does not contain any invalid information and can be exported to generate the corresponding SQL logic file A without cleaning.

[0120] By traversing and querying the database's DBC data dictionary, all valid view nodes and their corresponding structure definition DDL statements are obtained, and the corresponding relational entity B (equivalent to a data file) is exported and generated.

[0121] Typically, supported SQL types include Teradata, MySQL, GaussDB, and Hive, but there are no specific limitations on the supported SQL types. After the SQL statement is parsed by Antlr4 to generate an Unresolved Logical Plan, the Analyzer binds to the Catalog to generate a Logical Plan. The lineage parser analyzes the Logical Plan and performs different processing depending on the type of Logical Plan, such as recursive calls, analysis of source-target table mapping relationships, or analysis of column-to-column mapping relationships. Finally, the lineage parsing results are combined and returned to the caller.

[0122] Enable the Antlr4 program, which is compatible with Teradata and GaussDB syntax, input SQL file A and relational entity B, and generate the relevant abstract syntax tree to represent the data logic structure of each statement.

[0123] Example SQL is as follows:

[0124]

[0125]

[0126] In this embodiment, an abstract syntax tree is constructed based on the script file of the target data warehouse platform; the mapping relationship between adjacent upstream and downstream fields is obtained according to the abstract syntax tree, and a field mapping table is constructed based on the mapping relationship. This avoids the problems of lag in lineage analysis, daily batch date limitations, and complex log cleaning due to non-standardized logs, thus improving the effectiveness of information.

[0127] In one embodiment, a field-level lineage resolution method, such as Figure 4 As shown, it includes:

[0128] An abstract syntax tree is constructed based on the script files of the target data warehouse platform; the mapping relationship between adjacent upstream and downstream fields is obtained based on the abstract syntax tree, and a field mapping table is constructed based on the mapping relationship.

[0129] Retrieve query information; the query information includes the field to be queried, as well as the corresponding query type and query path. Query types are divided into first query type and second query type, and query paths are divided into upstream link query and downstream link query.

[0130] If the field to be queried is obtained, and the query type corresponding to the field is the first query type, the mapping relationship of the first mapping type in the field mapping table is used as the target mapping relationship; if the query type corresponding to the field is the second query type, the mapping relationship of the first mapping type and the mapping relationship of the second mapping type in the field mapping table are used as the target mapping relationship. The first mapping type is used to indicate that there are no restrictions on the mapping relationship between the upstream and downstream fields; the second mapping type is used to indicate that there are restrictions on the mapping relationship between the upstream and downstream fields. The field mapping table includes multiple fields and the mapping relationships between each field; the mapping relationships are used to represent the field processing information between the upstream and downstream fields.

[0131] From the field mapping table, determine the upstream and downstream fields corresponding to the target mapping relationship, and use the upstream and downstream fields corresponding to the target mapping relationship as the target fields.

[0132] Based on the mapping relationships between target fields, determine the upstream and downstream relationships between them. If a field does not have a corresponding upstream field but has a corresponding downstream field, use that field as the starting field. Generate a starting node based on the starting field. Based on the upstream and downstream relationships between target fields, starting from at least one starting node, generate field nodes that correspond one-to-one with each target field. If the mapping relationship between two field nodes is of the first mapping type, generate a first-representation relationship connection between the two fields. The first representation indicates that there are no restrictions on the mapping relationship between the two field nodes. If the mapping relationship between two field nodes is of the second mapping type, generate a second-representation relationship connection between the two fields. This yields a full-link field dependency graph (either a data flow full-link field dependency graph containing only the first mapping type or a restricted flow full-link field dependency graph containing both the first and second mapping types). The second representation indicates that there are restrictions on the mapping relationship between the two field nodes. The relationship connection in the first and second representations points from the upstream field node to the downstream field node. The full-link field dependency graph includes field nodes that correspond one-to-one with the target fields, and relationship connections between the field nodes representing the mapping relationships.

[0133] Based on the query field, the corresponding query field node is determined from the full-link field dependency graph. Then, based on the mapping relationships between field nodes in the full-link field dependency graph, the upstream and downstream field nodes corresponding to the query field are determined. If the query path corresponding to the query field is an upstream link query, the upstream field node corresponding to the query field is obtained from the full-link field dependency graph as the relevant field node; if the query path corresponding to the query field is a downstream link query, the downstream field node of the query field node is obtained from the full-link field dependency graph as the relevant field node. Based on the query field node and the relevant field nodes, the query field link is obtained from the full-link field dependency graph. The lineage analysis results of the query field are obtained based on the query field link.

[0134] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0135] Based on the same inventive concept, this application also provides a field-level lineage resolution apparatus for implementing the field-level lineage resolution method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more field-level lineage resolution apparatus embodiments provided below can be found in the limitations of the field-level lineage resolution method described above, and will not be repeated here.

[0136] In one embodiment, such as Figure 5 As shown, a field-level lineage resolution device 500 is provided, including: an acquisition module 501, a query module 502, a generation module 503, and a filtering module 504, wherein:

[0137] The acquisition module 501 is used to determine the target mapping relationship that matches the query type from the field mapping table if a field to be queried is acquired, based on the query type corresponding to the field to be queried. The field mapping table includes multiple fields and the mapping relationship between each field. The mapping relationship is used to represent the field processing information between upstream fields and downstream fields.

[0138] The query module 502 is used to determine at least one target field from the field mapping table based on the target mapping relationship;

[0139] The generation module 503 is used to generate a full-link field dependency graph based on the target fields and the mapping relationship between each target field. The full-link field dependency graph includes field nodes that correspond one-to-one with the target fields, and relationship lines between each field node to represent the mapping relationship.

[0140] The filtering module 504 is used to determine the corresponding query field node from the full-link field dependency graph based on the query field, obtain the query field link from the full-link field dependency graph based on the query field node, and obtain the lineage analysis result of the query field based on the query field link.

[0141] In one embodiment, the query type is divided into a first query type and a second query type. The acquisition module 501 is further configured to, if the query type corresponding to the field to be queried is the first query type, use the mapping relationship of the first mapping type in the field mapping table as the target mapping relationship; if the query type corresponding to the field to be queried is the second query type, use the mapping relationship of the first mapping type and the mapping relationship of the second mapping type in the field mapping table as the target mapping relationship; the first mapping type is used to indicate that there are no restrictions on the mapping relationship between the upstream field and the downstream field; the second mapping type is used to indicate that there are restrictions on the mapping relationship between the upstream field and the downstream field.

[0142] In one embodiment, the query module 502 is further configured to determine the upstream and downstream fields corresponding to the target mapping relationship from the field mapping table, and use the upstream and downstream fields corresponding to the target mapping relationship as the target fields.

[0143] In one embodiment, the generation module 503 is further configured to determine the upstream and downstream relationships between the target fields based on the mapping relationship between the target fields; determine at least one starting node of the full-link field dependency graph based on the upstream and downstream relationships between the target fields; generate field nodes corresponding one-to-one with each target field starting from at least one starting node based on the upstream and downstream relationships between the target fields, and generate relationship lines between the field nodes that match the corresponding mapping relationship to obtain the full-link field dependency graph.

[0144] In one embodiment, the generation module 503 is further configured to, if a field does not have a corresponding upstream field but has a corresponding downstream field, use the field as a starting field and generate a starting node based on the starting field.

[0145] In one embodiment, the generation module 503 is further configured to generate a first-form relationship connection between the two fields if the mapping relationship between the two field nodes is a first mapping type; the first form is used to represent that there are no restrictions on the mapping relationship between the two field nodes; if the mapping relationship between the two field nodes is a second mapping type, generate a second-form relationship connection between the two fields; the second form is used to represent that there are restrictions on the mapping relationship between the two field nodes; the first-form relationship connection and the second-form relationship connection point from the upstream field node to the downstream field node in the two field nodes.

[0146] In one embodiment, the filtering module 504 is further configured to determine the related field nodes corresponding to the query field node from the full-link field dependency graph based on the query path corresponding to the query field; and to obtain the query field link from the full-link field dependency graph based on the query field node and the related field nodes.

[0147] In one embodiment, the query path is divided into upstream link query and downstream link query. The filtering module 504 is also used to determine the upstream field node and downstream field node corresponding to the field to be queried based on the mapping relationship between each field node in the full link field dependency graph. If the query path corresponding to the field to be queried is an upstream link query, the upstream field node corresponding to the field to be queried is obtained from the full link field dependency graph as the relevant field node. If the query path corresponding to the field to be queried is a downstream link query, the downstream field node of the field to be queried is obtained from the full link field dependency graph as the relevant field node.

[0148] In one embodiment, the acquisition module 501 is further used to acquire query information; the query information includes the field to be queried, as well as the query type and query path corresponding to the field to be queried.

[0149] In one embodiment, the apparatus further includes:

[0150] The preprocessing module 505 is used to construct an abstract syntax tree based on the script file of the target data warehouse platform; obtain the mapping relationship between adjacent upstream and downstream fields according to the abstract syntax tree; and construct a field mapping table according to the mapping relationship.

[0151] Each module in the aforementioned field-level kinship analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0152] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a field-level lineage resolution method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0153] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0154] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: if a field to be queried is obtained, then, based on the query type corresponding to the field to be queried, determine a target mapping relationship matching the query type from a field mapping table; the field mapping table includes multiple fields and mapping relationships between each field; the mapping relationship is used to characterize field processing information between upstream and downstream fields; based on the target mapping relationship, determine at least one target field from the field mapping table; generate a full-link field dependency graph based on the target field and the mapping relationships between each target field; the full-link field dependency graph includes field nodes corresponding one-to-one with the target fields, and relationship lines between each field node used to represent mapping relationships; based on the field to be queried, determine the field node to be queried corresponding to the field to be queried from the full-link field dependency graph, and obtain the field link to be queried from the full-link field dependency graph based on the field node to be queried, and obtain the lineage analysis result of the field to be queried based on the field link to be queried.

[0155] In one embodiment, the query type is divided into a first query type and a second query type. When the processor executes the computer program, it further implements the following steps: if the query type corresponding to the field to be queried is the first query type, the mapping relationship of the first mapping type in the field mapping table is used as the target mapping relationship; if the query type corresponding to the field to be queried is the second query type, the mapping relationship of the first mapping type and the mapping relationship of the second mapping type in the field mapping table are used as the target mapping relationship; the first mapping type is used to represent that there are no restrictions on the mapping relationship between the upstream field and the downstream field; the second mapping type is used to represent that there are restrictions on the mapping relationship between the upstream field and the downstream field.

[0156] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the upstream and downstream fields corresponding to the target mapping relationship from the field mapping table, and using the upstream and downstream fields corresponding to the target mapping relationship as the target fields.

[0157] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the upstream and downstream relationships between each target field based on the mapping relationship between each target field; determining at least one starting node of the full-link field dependency graph based on the upstream and downstream relationships between each target field; generating field nodes that correspond one-to-one with each target field starting from at least one starting node based on the upstream and downstream relationships between each target field, and generating relationship connections between each field node that match the corresponding mapping relationship, thereby obtaining the full-link field dependency graph.

[0158] In one embodiment, when the processor executes the computer program, it further performs the following steps: if a field does not have a corresponding upstream field but has a corresponding downstream field, the field is used as the starting field; a starting node is generated based on the starting field.

[0159] In one embodiment, when the processor executes the computer program, it further performs the following steps: if the mapping relationship between two field nodes is a first mapping type, a relationship connection of a first representation form is generated between the two fields; the first representation form is used to characterize that there are no restrictions on the mapping relationship between the two field nodes; if the mapping relationship between two field nodes is a second mapping type, a relationship connection of a second representation form is generated between the two fields; the second representation form is used to characterize that there are restrictions on the mapping relationship between the two field nodes; the relationship connection of the first representation form and the relationship connection of the second representation form are from the upstream field node to the downstream field node in the two field nodes.

[0160] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the related field nodes corresponding to the query field node from the full-link field dependency graph based on the query path corresponding to the query field; and obtaining the query field link from the full-link field dependency graph based on the query field node and the related field nodes.

[0161] In one embodiment, the query path is divided into upstream link query and downstream link query. When the processor executes the computer program, it also performs the following steps: based on the mapping relationship between each field node in the full-link field dependency graph, determine the upstream field node and downstream field node corresponding to the field to be queried; if the query path corresponding to the field to be queried is an upstream link query, obtain the upstream field node corresponding to the field to be queried from the full-link field dependency graph as the relevant field node; if the query path corresponding to the field to be queried is a downstream link query, obtain the downstream field node of the field to be queried from the full-link field dependency graph as the relevant field node.

[0162] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining query information; the query information includes the field to be queried, as well as the query type and query path corresponding to the field to be queried.

[0163] In one embodiment, when the processor executes the computer program, it also performs the following steps: constructing an abstract syntax tree based on the script file of the target data warehouse platform; obtaining the mapping relationship between adjacent upstream fields and downstream fields according to the abstract syntax tree; and constructing a field mapping table according to the mapping relationship.

[0164] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: if a field to be queried is obtained, a target mapping relationship matching the query type is determined from a field mapping table according to the query type corresponding to the field to be queried; the field mapping table includes multiple fields and mapping relationships between the fields; the mapping relationship is used to characterize field processing information between upstream and downstream fields; at least one target field is determined from the field mapping table according to the target mapping relationship; a full-link field dependency graph is generated according to the target field and the mapping relationships between the target fields; the full-link field dependency graph includes field nodes that correspond one-to-one with the target fields and relationship lines between the field nodes used to represent the mapping relationship; according to the field to be queried, the corresponding field node to be queried is determined from the full-link field dependency graph, and the field link to be queried is obtained from the full-link field dependency graph according to the field node to be queried, and the lineage analysis result of the field to be queried is obtained based on the field link to be queried.

[0165] In one embodiment, the query type is divided into a first query type and a second query type. When the computer program is executed by the processor, it further implements the following steps: if the query type corresponding to the field to be queried is the first query type, the mapping relationship of the first mapping type in the field mapping table is used as the target mapping relationship; if the query type corresponding to the field to be queried is the second query type, the mapping relationship of the first mapping type and the mapping relationship of the second mapping type in the field mapping table are used as the target mapping relationship; the first mapping type is used to represent that there are no restrictions on the mapping relationship between the upstream field and the downstream field; the second mapping type is used to represent that there are restrictions on the mapping relationship between the upstream field and the downstream field.

[0166] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the upstream and downstream fields corresponding to the target mapping relationship from the field mapping table, and using the upstream and downstream fields corresponding to the target mapping relationship as the target fields.

[0167] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the upstream and downstream relationships between each target field based on the mapping relationship between each target field; determining at least one starting node of the full-link field dependency graph based on the upstream and downstream relationships between each target field; generating field nodes that correspond one-to-one with each target field starting from at least one starting node based on the upstream and downstream relationships between each target field, and generating relationship connections between each field node that match the corresponding mapping relationship, thereby obtaining the full-link field dependency graph.

[0168] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if a field does not have a corresponding upstream field but has a corresponding downstream field, the field is used as the starting field; a starting node is generated based on the starting field.

[0169] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the mapping relationship between two field nodes is a first mapping type, a relationship connection of a first representation form is generated between the two fields; the first representation form is used to characterize that there are no restrictions on the mapping relationship between the two field nodes; if the mapping relationship between two field nodes is a second mapping type, a relationship connection of a second representation form is generated between the two fields; the second representation form is used to characterize that there are restrictions on the mapping relationship between the two field nodes; the relationship connection of the first representation form and the relationship connection of the second representation form are from the upstream field node to the downstream field node in the two field nodes.

[0170] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the related field nodes corresponding to the query field node from the full-link field dependency graph based on the query path corresponding to the query field; and obtaining the query field link from the full-link field dependency graph based on the query field node and the related field nodes.

[0171] In one embodiment, the query path is divided into upstream link query and downstream link query. When the computer program is executed by the processor, it also performs the following steps: based on the mapping relationship between each field node in the full-link field dependency graph, determine the upstream field node and downstream field node corresponding to the field to be queried; if the query path corresponding to the field to be queried is an upstream link query, obtain the upstream field node corresponding to the field to be queried from the full-link field dependency graph as the relevant field node; if the query path corresponding to the field to be queried is a downstream link query, obtain the downstream field node of the field to be queried from the full-link field dependency graph as the relevant field node.

[0172] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining query information; the query information includes the field to be queried, as well as the query type and query path corresponding to the field to be queried.

[0173] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: constructing an abstract syntax tree based on the script file of the target data warehouse platform; obtaining the mapping relationship between adjacent upstream fields and downstream fields according to the abstract syntax tree; and constructing a field mapping table according to the mapping relationship.

[0174] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0175] If a field to be queried is obtained, the target mapping relationship matching the query type is determined from the field mapping table based on the query type corresponding to the field to be queried. The field mapping table includes multiple fields and the mapping relationships between each field. The mapping relationship is used to represent the field processing information between upstream and downstream fields. At least one target field is determined from the field mapping table based on the target field. A full-link field dependency graph is generated based on the target field and the mapping relationships between each target field. The full-link field dependency graph includes field nodes that correspond one-to-one with the target fields and the relationship lines between each field node used to represent the mapping relationship. Based on the field to be queried, the corresponding field node to be queried is determined from the full-link field dependency graph, and the field link to be queried is obtained from the full-link field dependency graph based on the field link to be queried. The lineage analysis result of the field to be queried is obtained based on the field link to be queried.

[0176] In one embodiment, the query type is divided into a first query type and a second query type. When the computer program is executed by the processor, it further implements the following steps: if the query type corresponding to the field to be queried is the first query type, the mapping relationship of the first mapping type in the field mapping table is used as the target mapping relationship; if the query type corresponding to the field to be queried is the second query type, the mapping relationship of the first mapping type and the mapping relationship of the second mapping type in the field mapping table are used as the target mapping relationship; the first mapping type is used to represent that there are no restrictions on the mapping relationship between the upstream field and the downstream field; the second mapping type is used to represent that there are restrictions on the mapping relationship between the upstream field and the downstream field.

[0177] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the upstream and downstream fields corresponding to the target mapping relationship from the field mapping table, and using the upstream and downstream fields corresponding to the target mapping relationship as the target fields.

[0178] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the upstream and downstream relationships between each target field based on the mapping relationship between each target field; determining at least one starting node of the full-link field dependency graph based on the upstream and downstream relationships between each target field; generating field nodes that correspond one-to-one with each target field starting from at least one starting node based on the upstream and downstream relationships between each target field, and generating relationship connections between each field node that match the corresponding mapping relationship, thereby obtaining the full-link field dependency graph.

[0179] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if a field does not have a corresponding upstream field but has a corresponding downstream field, the field is used as the starting field; a starting node is generated based on the starting field.

[0180] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the mapping relationship between two field nodes is a first mapping type, a relationship connection of a first representation form is generated between the two fields; the first representation form is used to characterize that there are no restrictions on the mapping relationship between the two field nodes; if the mapping relationship between two field nodes is a second mapping type, a relationship connection of a second representation form is generated between the two fields; the second representation form is used to characterize that there are restrictions on the mapping relationship between the two field nodes; the relationship connection of the first representation form and the relationship connection of the second representation form are from the upstream field node to the downstream field node in the two field nodes.

[0181] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the related field nodes corresponding to the query field node from the full-link field dependency graph based on the query path corresponding to the query field; and obtaining the query field link from the full-link field dependency graph based on the query field node and the related field nodes.

[0182] In one embodiment, the query path is divided into upstream link query and downstream link query. When the computer program is executed by the processor, it also performs the following steps: based on the mapping relationship between each field node in the full-link field dependency graph, determine the upstream field node and downstream field node corresponding to the field to be queried; if the query path corresponding to the field to be queried is an upstream link query, obtain the upstream field node corresponding to the field to be queried from the full-link field dependency graph as the relevant field node; if the query path corresponding to the field to be queried is a downstream link query, obtain the downstream field node of the field to be queried from the full-link field dependency graph as the relevant field node.

[0183] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining query information; the query information includes the field to be queried, as well as the query type and query path corresponding to the field to be queried.

[0184] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: constructing an abstract syntax tree based on the script file of the target data warehouse platform; obtaining the mapping relationship between adjacent upstream fields and downstream fields according to the abstract syntax tree; and constructing a field mapping table according to the mapping relationship.

[0185] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A field-level lineage resolution method, characterized in that, The method includes: If a field to be queried is obtained, then based on the query type corresponding to the field to be queried, a target mapping relationship matching the query type is determined from the field mapping table; the field mapping table includes multiple fields and the mapping relationship between each field; the mapping relationship is used to represent the field processing information between upstream fields and downstream fields; Based on the target mapping relationship, at least one target field is determined from the field mapping table; A full-link field dependency graph is generated based on the target fields and the mapping relationships between them; the full-link field dependency graph includes field nodes that correspond one-to-one with the target fields, and relationship lines between the field nodes to represent the mapping relationships; Based on the field to be queried, determine the field node corresponding to the field to be queried from the full-link field dependency graph, obtain the field link to be queried from the full-link field dependency graph based on the field node to be queried, and obtain the lineage analysis result of the field to be queried based on the field link to be queried; The query types are divided into a first query type and a second query type; determining the target mapping relationship matching the query type from the field mapping table according to the query type corresponding to the field to be queried includes: If the query type corresponding to the field to be queried is the first query type, the mapping relationship of the first mapping type in the field mapping table is used as the target mapping relationship; if the query type corresponding to the field to be queried is the second query type, the mapping relationship of the first mapping type and the mapping relationship of the second mapping type in the field mapping table are used as the target mapping relationship; the first mapping type is used to indicate that there are no restrictions on the mapping relationship between the upstream field and the downstream field; the second mapping type is used to indicate that there are restrictions on the mapping relationship between the upstream field and the downstream field.

2. The method according to claim 1, characterized in that, The step of determining the target field from the field mapping table based on the target mapping relationship includes: From the field mapping table, determine the upstream and downstream fields corresponding to the target mapping relationship, and use the upstream and downstream fields corresponding to the target mapping relationship as the target fields.

3. The method according to claim 1, characterized in that, The step of generating a full-link field dependency graph based on the target fields and the mapping relationships between them includes: Based on the mapping relationship between each target field, determine the upstream and downstream relationships between each target field; Based on the upstream and downstream relationships between each target field, at least one starting node of the full-link field dependency graph is determined; Based on the upstream and downstream relationships between each target field, starting from at least one starting node, field nodes corresponding one-to-one with each target field are generated, and relationship lines matching the corresponding mapping relationships are generated between each field node to obtain the full-link field dependency graph.

4. The method according to claim 3, characterized in that, The step of determining at least one starting node of the full-link field dependency graph based on the upstream and downstream relationships between each field includes: If a field does not have a corresponding upstream field, but has a corresponding downstream field, then the field is used as the starting field. A starting node is generated based on the starting field.

5. The method according to claim 3, characterized in that, The step of generating relationship connections between each field node that match the corresponding mapping relationship includes: If the mapping relationship between two field nodes is of the first mapping type, a relationship connection of the first representation form is generated between the two fields; the first representation form is used to characterize that there are no restrictive conditions in the mapping relationship between the two field nodes. If the mapping relationship between two field nodes is of the second mapping type, a relationship connection with a second representation is generated between the two fields; the second representation is used to characterize that there are restrictions on the mapping relationship between the two field nodes. The relationship connection in the first representation and the relationship connection in the second representation are connected by the upstream field node pointing to the downstream field node in the two field nodes.

6. The method according to claim 1, characterized in that, The step of obtaining the query field link from the full-link field dependency graph based on the query field node includes: Based on the query path corresponding to the field to be queried, determine the relevant field nodes corresponding to the node of the field to be queried from the full-link field dependency graph; Based on the field node to be queried and the related field nodes, the link of the field to be queried is obtained from the full-link field dependency graph.

7. The method according to claim 6, characterized in that, The query path is divided into upstream link query and downstream link query. The step of determining the relevant field nodes corresponding to the node of the field to be queried from the full-link field dependency graph based on the query path corresponding to the field to be queried includes: Based on the mapping relationship between each field node in the full-link field dependency graph, determine the upstream field node and downstream field node corresponding to the field to be queried; If the query path corresponding to the field to be queried is the upstream link query, the upstream field node corresponding to the field to be queried is obtained from the full link field dependency graph as the relevant field node; If the query path corresponding to the field to be queried is the downstream link query, the downstream field node of the field to be queried is obtained from the full-link field dependency graph as the related field node.

8. The method according to claim 1, characterized in that, Before determining the target mapping relationship matching the query type from the field mapping table based on the query type corresponding to the obtained query field, the process further includes: Obtain query information; the query information includes the field to be queried, as well as the query type and query path corresponding to the field to be queried.

9. The method according to claim 1, characterized in that, The method further includes: Construct an abstract syntax tree based on script files from the target data warehouse platform; The mapping relationship between adjacent upstream and downstream fields is obtained based on the abstract syntax tree, and the field mapping table is constructed based on the mapping relationship.

10. A field-level lineage analysis device, characterized in that, The device includes: The acquisition module is used to, if a field to be queried is acquired, determine a target mapping relationship matching the query type from a field mapping table based on the query type corresponding to the field to be queried; the field mapping table includes multiple fields and mapping relationships between each field; the mapping relationship is used to represent field processing information between upstream fields and downstream fields; The query module is used to determine at least one target field from the field mapping table based on the target mapping relationship; The generation module is used to generate a full-link field dependency graph based on the target fields and the mapping relationships between the target fields; the full-link field dependency graph includes field nodes that correspond one-to-one with the target fields, and relationship lines between the field nodes to represent the mapping relationships; The filtering module is used to determine the query field node corresponding to the query field from the full-link field dependency graph based on the query field, obtain the query field link from the full-link field dependency graph based on the query field node, and obtain the lineage analysis result of the query field based on the query field link; The query types are divided into a first query type and a second query type; determining the target mapping relationship matching the query type from the field mapping table according to the query type corresponding to the field to be queried includes: If the query type corresponding to the field to be queried is the first query type, the mapping relationship of the first mapping type in the field mapping table is used as the target mapping relationship; if the query type corresponding to the field to be queried is the second query type, the mapping relationship of the first mapping type and the mapping relationship of the second mapping type in the field mapping table are used as the target mapping relationship; the first mapping type is used to indicate that there are no restrictions on the mapping relationship between the upstream field and the downstream field; the second mapping type is used to indicate that there are restrictions on the mapping relationship between the upstream field and the downstream field.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images