Data blood relationship determination method and device, storage medium and electronic equipment

Abstract

The invention discloses a data consanguinity determination method and device, a storage medium and electronic equipment, and the method comprises the steps: recording a program file corresponding to an operation code of a data processing operation of a data processing link in each data processing link of a target business, and recording the position of the operation code in the program file; and determining a program index according to the recorded program file corresponding to the operation code and the position of the operation code in the program file. According to the program index of each data processing link, index information of table nodes corresponding to the result data of each data processing link is determined, and the blood relationship link diagram is determined according to each table node, edges between the table nodes and the index information of each table node. According to the embodiment of the invention, the index information is contained in the blood relationship link diagram, so that when the data is abnormal, the position of the operation code of the data processing operation of the abnormal result data can be quickly determined in the program code according to the index information in the blood relationship link diagram, and the abnormality checking efficiency is accelerated.

Description

Technical Field

[0001] The present specification relates to the field of computer technology, and in particular to a method, device, storage medium and electronic device for determining data lineage. Background Art

[0002] Data lineage refers to the complete process of data changes during its life cycle, including each link in the data life cycle such as origin, transmission, storage, processing and use. It is the data link between data processing operations in multiple links from the initial generation to the final destination of the data.

[0003] Usually, when the result data obtained by the data processing operation of a certain link is abnormal, it is necessary to trace the upstream data of the abnormal data through the data link to find out the cause of the abnormality. The current method of investigation is mainly to check the operation code of the data processing operation of this link. However, since the code volume of the program code used for business data processing is generally very large, it is difficult to locate the operation code of the data processing operation of this link in the program code used for business data processing, and the efficiency of exception handling is low.

[0004] Therefore, this specification provides a method for determining data lineage. Summary of the invention

[0005] This specification provides a data lineage determination method, device, storage medium and electronic device to at least partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a method for determining data lineage, including:

[0008] Determine a target business, and in each data processing link of the target business, record a program file corresponding to an operation code of a data processing operation of the data processing link, and record a position of the operation code in the program file;

[0009] Determine the program index of the data processing link according to the program file corresponding to the recorded operation code and the position of the operation code in the program file;

[0010] Determine the original data of the target business, and determine the result data obtained by each data processing link, and determine the data flow direction between the original data and each result data according to the operation code of the data processing operation of each data processing link;

[0011] Determine the nodes corresponding to the original data and the nodes corresponding to the result data as table nodes, determine the edges between the table nodes according to the data flow direction between the original data and the result data, and determine the index information of the table nodes corresponding to the result data of each data processing link according to the program index of each data processing link;

[0012] A blood relationship link graph is determined according to the table nodes, the edges between the table nodes and the index information of the table nodes.

[0013] Optionally, the operation code includes a program statement for information reporting;

[0014] According to the operation code of the data processing operation of each data processing link, determining the data flow direction between the original data and each result data specifically includes:

[0015] For each data processing link, according to the operation code of the data processing operation of the data processing link, the original data and each result data used in the data processing operation of the data processing link are obtained as the source data of the data processing link;

[0016] Determine the data flow direction from each source data to the result data obtained in the data processing link.

[0017] Optionally, determining the data flow direction between the original data and each result data according to the operation code of the data processing operation of each data processing link specifically includes:

[0018] For each data processing link, determine the syntax tree corresponding to the operation code of the data processing operation of the data processing link;

[0019] Each node of the syntax tree is traversed, and the data flow direction corresponding to the data processing link is determined according to the nodes representing the query field and the nodes representing the modification field in the syntax tree.

[0020] Optionally, determining the data flow direction corresponding to the data processing link according to the node representing the query field and the node representing the modification field in the syntax tree specifically includes:

[0021] In each node of the syntax tree, the source data of the data processing link is determined according to the child nodes of the node representing the query field, and the result data obtained by the data processing link is determined according to the child nodes of the node representing the modification field;

[0022] The data flow direction from each source data to the result data obtained in the data processing link is determined respectively.

[0023] Optionally, determining index information of table node connections corresponding to result data of each data processing link according to the program index of each data processing link respectively includes:

[0024] For each data processing link, according to the determined data flow direction between the original data and the result data, determine the original data and the result data located upstream of the result data of the data processing link as the upstream data of the result data of the data processing link;

[0025] According to the data flow direction between the original data and the result data, and the data flow direction between the result data of the data processing link and the upstream data, the program indexes of the result data of the data processing link and the upstream data are sorted to determine the index chain;

[0026] According to the index chain, the index information of the table node connection corresponding to the result data of the data processing link is determined.

[0027] Optionally, the method further comprises:

[0028] Using the program index of the data processing link in the index chain as the main information;

[0029] Using the index chain as folding information of the main information;

[0030] In response to a user's abnormal query command for the result data of the data processing link, determine a table node corresponding to the result data of the data processing link, and display the main information;

[0031] In response to the user's command to expand the main information, the collapsed information of the main information is displayed.

[0032] Optionally, the blood relationship link graph further includes column nodes;

[0033] The method further comprises:

[0034] For each table node in the blood relationship link graph, determine each column node connected to the table node according to each data column included in the original data or result data corresponding to the table node;

[0035] For each data column included in the original data or result data corresponding to the table node, determine a program statement corresponding to the data column in the operation code of the data processing operation corresponding to the table node;

[0036] Determine, according to the program statement corresponding to the data column, a data processing rule corresponding to the data column, and determine the data column used by the data processing rule as a source column;

[0037] An edge from the source column to the data column is determined.

[0038] This specification provides a data lineage determination device, the device comprising:

[0039] A recording module determines a target business, and in each data processing link of the target business, records a program file corresponding to an operation code of a data processing operation of the data processing link, and records a position of the operation code in the program file;

[0040] A program index determination module, which determines the program index of the data processing link according to the program file corresponding to the recorded operation code and the position of the operation code in the program file;

[0041] A data flow direction determination module determines the original data of the target business and the result data obtained in each data processing link, and determines the data flow direction between the original data and each result data according to the operation code of the data processing operation of each data processing link;

[0042] A construction module is provided to determine the nodes corresponding to the original data and the nodes corresponding to the result data as table nodes, determine the edges between the table nodes according to the data flow direction between the original data and the result data, and determine the index information of the table nodes corresponding to the result data of each data processing link according to the program index of each data processing link;

[0043] The data lineage determination module determines a lineage link graph according to the table nodes, the edges between the table nodes and the index information of the table nodes.

[0044] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data lineage determination method is implemented.

[0045] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data lineage determination method when executing the program.

[0046] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0047] In the data lineage determination method provided in this specification, in each data processing link of the target business, the program file corresponding to the operation code of the data processing operation of the data processing link is recorded, and the position of the operation code in the program file is recorded. According to the program file corresponding to the recorded operation code and the position of the operation code in the program file, the program index of the data processing link is determined. The nodes corresponding to the original data of the target business and the nodes corresponding to each result data are determined as table nodes. According to the data flow direction between the original data and each result data, the edges between each table node are determined. According to the program index of each data processing link, the index information of the table node corresponding to the result data of each data processing link is determined. According to each table node, the edges between each table node and the index information of each table node, the lineage link graph is determined. In the lineage link graph obtained by this method, because it contains index information, when the data is abnormal, the position of the operation code of the data processing operation of the abnormal result data can be quickly determined in the program code according to the index information in the lineage link graph, thereby speeding up the efficiency of abnormal troubleshooting. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0049] Figure 1 A flowchart of a method for determining data lineage in this specification;

[0050] Figure 2 A schematic diagram of a blood relationship link diagram provided in an embodiment of this specification;

[0051] Figure 3 The corresponding Figure 2 A schematic diagram of another type of bloodline link diagram;

[0052] Figure 4 A schematic diagram of a data lineage determination device provided in this specification;

[0053] Figure 5 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0056] Figure 1 The following is a flow chart of a method for determining data lineage in this specification, which specifically includes the following steps:

[0057] S100: Determine a target business, and in each data processing link of the target business, record a program file corresponding to an operation code of a data processing operation of the data processing link, and record a position of the operation code in the program file.

[0058] All steps in the data lineage determination method provided in this specification can be implemented by any electronic device with computing functions, such as a terminal, a server, etc. For ease of description, the data lineage determination method provided in this specification is described below using only the server as the execution subject.

[0059] In order to locate the operation code of the data processing operation of each data processing link after each data processing link is completed, the server determines the target business, and in each data processing link of the target business, that is, when the user creates the operation code of the data processing link, records the program file corresponding to the operation code of the data processing operation of the data processing link, and records the location of the operation code in the program file.

[0060] Because the operation codes of different data processing links are usually saved in different program files, recording the program file corresponding to the operation code can facilitate the rapid positioning of the operation code in the file range in subsequent exception analysis, determine the program file where the operation code is located, and then locate the precise position of the operation code based on the position of the operation code in the program file.

[0061] Usually, there is a hierarchical dependency between the various data processing links, that is, for a certain data processing link, if the data processing link needs to use the result data of other data processing links, then the result data of the data processing link is the downstream data of the result data of other data processing links that the data processing link needs to use, and the result data of other data processing links that the data processing link needs to use are the upstream data of the result data of the data processing link.

[0062] S102: Determine the program index of the data processing link according to the recorded program file corresponding to the operation code and the position of the operation code in the program file.

[0063] In order to enable users to quickly find the operation code of the data processing operation corresponding to the abnormal data in a large amount of program code when discovering data anomalies, the server determines the program index of the data processing link based on the program file corresponding to the recorded operation code and the position of the operation code in the program file.

[0064] The program index is used to locate the position of the operation code in all program codes corresponding to the target business. This specification does not limit the specific form of the program index. For example, the program index can be composed of a program file identifier and the line number of the first line of each program statement included in the operation code in the program file, or it can be composed of a program file identifier, the line number of the first line of each program statement included in the operation code in the program file, and the line number of the last line of each program statement included in the operation code in the program file.

[0065] S104: Determine the original data of the target business and the result data obtained in each data processing link, and determine the data flow direction between the original data and each result data according to the operation code of the data processing operation of each data processing link.

[0066] Raw data is data directly obtained by the server without data processing. Raw data is the basis for determining data lineage. This manual does not limit the source of raw data. It can be the behavioral data of front-end users, such as data generated by user clicks, inputs, browsing, etc., or the device data of the business system deployment device corresponding to the target business of the server, such as device model, logs, etc. It can also be the operation data generated by the server during operation, product information data, etc.

[0067] The server can obtain raw data from one or more sources for data lineage construction. For example, the server can obtain user information tables, user front-end behavior tables, and product tables as raw data.

[0068] The result data is the data obtained after a data processing operation in a data processing link. The server needs to determine the data flow direction between the original data and each result data, so that the original data and each result data can be associated in the lineage link diagram.

[0069] There are many methods for determining the direction of data flow. In one or more embodiments of this specification, the operation code includes a program statement for information reporting, through which the original data and each result data used in the data processing operation can be sent to the server.

[0070] Therefore, for each data processing link, the server can obtain the original data and each result data used in the data processing operation of the data processing link according to the operation code of the data processing operation of the data processing link as the source data of the data processing link. Then, for each source data, the data flow direction from the source data to the result data obtained by the data processing link is determined.

[0071] S106: Determine the nodes corresponding to the original data and the nodes corresponding to each result data as table nodes, determine the edges between the table nodes according to the data flow direction between the original data and the result data, and determine the index information of the table nodes corresponding to the result data of each data processing link according to the program index of each data processing link.

[0072] The bloodline link graph constructed in this specification may include table nodes, index nodes, and edges between table nodes.

[0073] First, the server determines the node corresponding to the original data and the node corresponding to each result data as the table node. In the lineage link graph, each original data or result data has a corresponding table node.

[0074] Secondly, the server determines the edges between each table node according to the data flow direction between the original data and each result data. The edges in this specification are directed edges, which are used to characterize the data inflow and outflow directions of the original data and each result data. For a table node, the table node can be connected to one or more edges in the outflow direction, and another table node corresponding to the edge in the outflow direction connected to the table node is the downstream node of the table node. The table node can be connected to one or more edges in the inflow direction, and another table node corresponding to the edge in the inflow direction connected to the table node is the upstream node of the table node. That is, the downstream node of the table node is the table node of the downstream data of the result data corresponding to the table node, and the upstream node of the table node is the table node of the upstream data of the result data corresponding to the table node.

[0075] Then, the server determines the index information of the table node corresponding to the result data of each data processing link according to the program index of each data processing link. The index information is a kind of attribute information of the table node corresponding to the result data of the data processing link.

[0076] This specification does not limit the specific display form of the index information. The index information can be displayed in the form of nodes, or directly in the form of text information, etc. When the index information is displayed in the form of nodes, the server generates nodes corresponding to each index information in the bloodline link diagram as index nodes. In the bloodline link diagram, an index node corresponding to a table node is connected to the table node. When the index information is displayed in the form of text information, the server can display the text content of the index information corresponding to each table node around each table node in the bloodline link diagram.

[0077] S108: Determine a blood relationship link graph according to the table nodes, the edges between the table nodes and the index information of the table nodes.

[0078] After the server determines the table nodes, the edges between the table nodes, and the index information of the table nodes, the blood relationship link graph in this specification can be obtained.

[0079] Figure 2 Schematic diagram of a blood relationship link diagram provided in the embodiments of this specification. Figure 2 As shown in the figure, a large circle represents a table node, and a small circle represents an index node of index information. T0 represents the table node corresponding to the original data, and T1 to T5 represent the table nodes corresponding to the result data of different data processing links. The table nodes are connected by directed edges according to the direction of data flow. The index node connected to each table node is used to represent the index information of the data processing link corresponding to the table node.

[0080] Figure 2 The index information is in the form of the file identifier of the program file and the line number of the first line of each program statement contained in the operation code in the program file. Figure 2 In the , lowercase letters are used to identify the file identifier, and the numbers after the file identifier indicate the line number.

[0081] Taking the table node corresponding to T1 as an example, the index information of the index node connected to the table node is "c_7", which means that the file where the operation code of the data processing operation of the data processing link corresponding to T1 is located is c, and the location of the operation code is the 7th line in file c.

[0082] The lineage link diagram obtained by this method contains index information. When data abnormalities occur, the location of the operation code for the data processing operation of the abnormal result data can be quickly determined in the program code based on the index information in the lineage link diagram, thereby speeding up the efficiency of abnormality detection.

[0083] In the above step S104, the server may also determine the data flow direction according to the syntax tree of the operation code.

[0084] In one or more embodiments of the present specification, the server performs syntax analysis on the operation code of the data processing operation of each data processing link through a syntax parser to determine the syntax tree corresponding to the operation code of the data processing operation of the data processing link. Each node in the syntax tree represents various syntax structures in the database operation language, and the edges in the syntax tree represent the syntax hierarchical relationship between the syntax structures.

[0085] Then, the server traverses each node of the syntax tree, and determines the data flow direction corresponding to the data processing link according to the nodes representing the query field and the nodes representing the modification field in the syntax tree.

[0086] According to the grammatical rules of the database operation language, the data table connected to the fields representing the query is the original data or result data that needs to be processed in the data processing operation, and the data table connected to the fields representing the modification is the result data corresponding to the data processing operation. The fields representing the query and the fields representing the modification can be predetermined according to the grammatical rules of the database operation language. For example, the fields representing the query include the SELECT field, the FROM field, etc., and the fields representing the modification include the INSERT field, the UPSERT field, the CREATE field, etc.

[0087] Specifically, in each node of the syntax tree, the server determines the source data of the data processing link according to the child nodes of the node representing the query field, and determines the result data obtained by the data processing link according to the child nodes of the node representing the modified field, and determines the data flow direction from the source data to the result data obtained by the data processing link.

[0088] Usually, when abnormal data appears, it is necessary to check the operation codes corresponding to each upstream data of the abnormal data layer by layer according to the lineage link diagram to determine the cause of the abnormality. Then, in the abnormal analysis, after the user determines the location of the operation code of the abnormal program according to the program index of the abnormal data, it is also necessary to determine the index information corresponding to each upstream node of the abnormal data according to each upstream node of the table node corresponding to the abnormal data in turn, so as to determine the location of all the operation codes that need to be checked.

[0089] Although the location of the operation code can be quickly located based on the index information in the bloodline link diagram, in complex business scenarios, the data processing links are relatively complicated, resulting in a large number of upstream nodes for an abnormal data corresponding table node. It still takes a long time to check the index information of each upstream node one by one.

[0090] Therefore, in the above step S106, the server may also determine the index information in the following manner.

[0091] First, for each processing link, the server determines the original data and result data located upstream of the result data of the data processing link according to the data flow direction between the original data and the result data determined in step S104 as the upstream data of the result data of the processing link.

[0092] Then, according to the data flow direction between the original data and each result data, the result data of the data processing link and the program indexes of each upstream data are sorted to determine an index chain. The index chain is a link composed of the program indexes of the result data and each upstream data.

[0093] Since one or more data flow paths may be determined based on the complexity of the data processing link and the data flow direction between the result data of the data processing link and each upstream data, when the server determines multiple data flow paths, the index chain corresponding to each data flow path may be determined respectively.

[0094] Finally, the server determines the index information of the table node connection corresponding to the result data according to the index chain. The index information may include one or more index chains corresponding to the data flow path.

[0095] In this embodiment, the index information of the table node connection corresponding to the result data of the data processing link represents an index chain composed of program indexes corresponding to the table node and all upstream nodes of the table node.

[0096] Figure 3 The corresponding Figure 2 A schematic diagram of another bloodline link diagram, Figure 3 The index information represented by the index node in the bloodline link diagram shown is the index chain corresponding to the table node.

[0097] like Figure 3 As shown, the index node of the table node corresponding to T5 represents the index information "a_5→b_6→c_30→e_20", and the index node of the table node corresponding to T4 represents the index information "a_5→b_6→c_30". It can be seen that in the bloodline link diagram, the index chain corresponding to the table node that is further downstream in the data flow direction is longer.

[0098] When an exception occurs in the result data, the user only needs to check the table node connection index node through the bloodline link diagram to obtain the location of the operation code of the data processing operation corresponding to all upstream nodes of the table node. There is no need to check the index information of each upstream node, which simplifies the user's operation steps in exception analysis and speeds up the efficiency of exception handling.

[0099] As mentioned above, in business scenarios, data processing is complex, and the number of table nodes included in the lineage link diagram is usually large, resulting in a long index chain. If the index information of each table node determined by the index chain is displayed in the lineage link diagram, the overall page of the lineage link diagram will be dense, which may make it difficult for users to identify the content they want to view.

[0100] In order to make the overall page of the bloodline diagram more concise, in one or more embodiments of the present specification, the server uses the program index of the data processing link as the main information, and uses the index chain composed of the program index of the result data of the data processing link and the upstream data of the result data of the data processing link as the folding information of the main information.

[0101] When displaying the bloodline link diagram to the user, the index node determined based on the main information is displayed first. At this time, the display effect of the bloodline link diagram in this embodiment is the same as the display effect of the bloodline link diagram determined in the above step S108. In response to the user's abnormal query command for the result data of the data processing link, the table node corresponding to the result data of the data processing link is determined, and the main information of the table node corresponding to the result data of the data processing link is displayed. Because the index node displayed at this time only represents a program index, the overall page is relatively concise, the user experience is better, and it is convenient to determine the content you want to view in the bloodline link diagram.

[0102] When the user needs to perform an exception analysis through the index node of the table node, the user can click on the index information of the table node, indicating that the user sends an expansion command for the main information to the server. The server responds to the user's expansion command for the main information and displays the collapsed information of the main information, that is, displays the index chain corresponding to the index node.

[0103] Through this embodiment, in the overall page of the bloodline link diagram, the index node corresponding to each table node displays the main information, that is, the program index of the data processing link corresponding to the table node is still displayed. The information is folded, that is, the index chain corresponding to the table node is displayed separately when the user needs to view it. This simplifies the display page of the bloodline link diagram and retains the index chain information in the index node, so that the user can obtain the location of the operation code corresponding to the upstream node of the table node only through the index node of the table node, thereby speeding up the abnormality analysis rate.

[0104] The bloodline link diagram in this specification is scalable. Users can add new data processing links according to business scenario requirements, and determine the table node of the result data corresponding to the newly added data processing link, as well as the edges between the table node and other table nodes in the original bloodline link diagram according to the above-mentioned methods S100 to S106, to update the bloodline link diagram.

[0105] In some business scenarios, different data processing links are handled by different users, so the same or similar data processing operations may be created in different data processing links. In the bloodline link diagram, the link information related to the table nodes corresponding to the result data of these same or similar data processing operations is highly repeated, resulting in information redundancy and increased storage pressure.

[0106] In order to reduce information redundancy, in one or more embodiments of this specification, the server obtains the data processing operation corresponding to the data processing link newly added by the user as the newly added operation. Then, the server determines the similarity between the data processing operation of the result data corresponding to each table node in the existing blood relationship link diagram and the newly added operation.

[0107] When the server determines that there is a similarity greater than a preset value among the similarities, it determines the data processing operation corresponding to the similarity greater than the preset value as the target operation, and generates prompt information based on the target operation, which is used to prompt the user to reuse the target operation.

[0108] The preset value in this embodiment can be set according to the requirements. If the similarity is greater than the preset value, it means that among the data processing operations created by the user in the historical data processing links, there is a data processing operation that is the same or similar to the data processing operation that the user is about to create, that is, the target operation. In this case, the server can push the target operation to the user through prompt information, reducing the information redundancy in the blood link diagram.

[0109] In order to make the bloodline link diagram in this specification more accurately represent the flow direction of data, in one or more embodiments of this specification, the bloodline link diagram may also include column nodes.

[0110] After the above step S106, the server determines, for each table node in the bloodline link diagram, the column nodes connected to the table node according to the data columns contained in the original data or result data corresponding to the table node.

[0111] Then, for each data column included in the original data or result data corresponding to the table node, the server determines the program statement corresponding to the data column in the operation code of the data processing operation of the original data or result data corresponding to the table node. According to the program statement corresponding to the data column, the server determines the data processing rule corresponding to the data column, and determines the data column used by the data processing rule as the source column. The edge from the source column to the data column is determined.

[0112] According to this embodiment, the server constructs a lineage link graph consisting of table nodes, column nodes, index nodes, edges between table nodes, and edges between column nodes. Based on the lineage link graph, users can perform table-level anomaly analysis and more accurate column-level anomaly analysis.

[0113] In one or more embodiments of the present specification, the bloodline diagram in the present specification can also be used for data heat analysis.

[0114] The server determines the usage popularity of the result data corresponding to each table node based on the number of edges flowing out from each table node in the bloodline link graph.

[0115] The higher the usage popularity of a result data, the more frequently the result data is used, and the higher the importance of the result data. The lower the usage popularity of a result data, the less frequently the result data is used, and the lower the importance of the result data.

[0116] The server determines for each table node whether the usage popularity of the result data corresponding to the table node is less than the preset lower limit. When the usage popularity of the result data corresponding to the table node is less than the preset lower limit, the result data corresponding to the table node is taken offline and no longer stored, thereby releasing the storage space corresponding to the result data.

[0117] Among them, when determining the usage heat, it can be intuitively displayed through visual icons, such as Sankey diagrams, heat maps, etc.

[0118] The above is the data lineage determination method provided in this specification. Based on the same idea, this specification also provides a corresponding data lineage determination device, such as Figure 4 shown.

[0119] Figure 4 A schematic diagram of a data lineage determination device provided in this specification specifically includes:

[0120] The recording module 200 is used to determine the target business, and in each data processing link of the target business, record the program file corresponding to the operation code of the data processing operation of the data processing link, and record the position of the operation code in the program file;

[0121] A program index determination module 202, configured to determine the program index of the data processing link according to the program file corresponding to the recorded operation code and the position of the operation code in the program file;

[0122] The data flow direction determination module 204 is used to determine the original data of the target business and the result data obtained by each data processing link, and determine the data flow direction between the original data and each result data according to the operation code of the data processing operation of each data processing link;

[0123] A construction module 206 is used to determine the nodes corresponding to the original data and the nodes corresponding to the result data as table nodes, determine the edges between the table nodes according to the data flow direction between the original data and the result data, and determine the index information of the table nodes corresponding to the result data of each data processing link according to the program index of each data processing link;

[0124] The data lineage determination module 208 is used to determine a lineage link graph based on the table nodes, the edges between the table nodes and the index information of the table nodes.

[0125] Optionally, the operation code includes program statements for information reporting, and the data flow direction determination module 204 is specifically used to obtain, for each data processing link, the original data and result data used in the data processing operation of the data processing link according to the operation code of the data processing operation of the data processing link, as the source data of the data processing link, and determine the data flow direction from each source data to the result data obtained from the data processing link.

[0126] Optionally, the data flow direction determination module 204 is specifically used to determine, for each data processing link, a syntax tree corresponding to an operation code of a data processing operation of the data processing link, traverse each node of the syntax tree, and determine the data flow direction corresponding to the data processing link based on the nodes representing query fields and the nodes representing modification fields in the syntax tree.

[0127] Optionally, the data flow direction determination module 204 is specifically used to determine the source data of the result data and the data flow direction from the source data to the result data in each node of the syntax tree according to the child nodes of the node representing the query field.

[0128] Optionally, the construction module 206 is specifically used to determine, for each data processing link, the original data and each result data located upstream of the result data of the data processing link according to the determined data flow direction between the original data and the result data, as the upstream data of the result data of the data processing link, and sort the program indexes of the result data of the data processing link and the upstream data according to the data flow direction between the original data and the result data and the data flow direction between the result data of the data processing link and the upstream data, determine the index chain, and determine the index information of the table node connection corresponding to the result data of the data processing link according to the index chain.

[0129] Optionally, the construction module 206 is specifically used to use the program index of the data processing link in the index chain as the main information, and use the index chain as the folding information of the main information, in response to the user's abnormal query command for the result data of the data processing link, determine the table node corresponding to the result data of the data processing link, display the main information, and in response to the user's expansion command for the main information, display the folding information of the main information.

[0130] Optionally, the bloodline link diagram also includes column nodes, and the construction module 206 is specifically used to determine, for each table node in the bloodline link diagram, the column nodes connected to the table node according to the data columns contained in the original data or result data corresponding to the table node, and for each data column contained in the original data or result data corresponding to the table node, determine the program statement corresponding to the data column in the operation code of the data processing operation corresponding to the table node, determine the data processing rule corresponding to the data column according to the program statement corresponding to the data column, and determine the data column used by the data processing rule as the source column, and determine the edge from the source column to the data column.

[0131] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The method for determining the lineage of the data provided.

[0132] This manual also provides Figure 5 The schematic structure diagram of the electronic device shown in FIG. Figure 5 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0133] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0134] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0135] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0136] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0137] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0138] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0141] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0142] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0143] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0144] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0145] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0147] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0148] The above description is only an embodiment of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this application.

Claims

1. A method for determining data lineage, characterized in that: include: Determine a target business, and in each data processing link of the target business, record a program file corresponding to an operation code of a data processing operation of the data processing link, and record a position of the operation code in the program file; Determine the program index of the data processing link according to the program file corresponding to the recorded operation code and the position of the operation code in the program file; Determine the original data of the target business and the result data obtained by each data processing link, and determine the data flow direction between the original data of the target business and the result data obtained by each data processing link according to the operation code of the data processing operation of each data processing link; Determine the nodes corresponding to the original data and the nodes corresponding to the result data as table nodes, determine the edges between the table nodes according to the data flow direction between the original data and the result data, and determine the index information of the table nodes corresponding to the result data of each data processing link according to the program index of each data processing link; A blood relationship link graph is determined according to the table nodes, the edges between the table nodes and the index information of the table nodes.

2. The method according to claim 1, characterized in that The operation code includes a program statement for information reporting; Determining the data flow direction between the original data and each result data according to the operation code of the data processing operation of each data processing link, specifically includes: For each data processing link, according to the operation code of the data processing operation of the data processing link, the original data and each result data used in the data processing operation of the data processing link are obtained as the source data of the data processing link; Determine the data flow direction from each source data to the result data obtained in the data processing link.

3. The method according to claim 1, characterized in that According to the operation code of the data processing operation of each data processing link, determining the data flow direction between the original data and each result data specifically includes: For each data processing link, determine the syntax tree corresponding to the operation code of the data processing operation of the data processing link; Each node of the syntax tree is traversed, and the data flow direction corresponding to the data processing link is determined according to the nodes representing the query field and the nodes representing the modification field in the syntax tree.

4. The method according to claim 3, characterized in that Determining the data flow direction corresponding to the data processing link according to the nodes representing the query field and the nodes representing the modified field in the syntax tree, specifically includes: In each node of the syntax tree, the source data of the data processing link is determined according to the child nodes of the node representing the query field, and the result data obtained by the data processing link is determined according to the child nodes of the node representing the modification field; The data flow direction from each source data to the result data obtained in the data processing link is determined respectively.

5. The method according to claim 1, characterized in that According to the program index of each data processing link, the index information of the table node connection corresponding to the result data of each data processing link is determined, specifically including: For each data processing link, according to the determined data flow direction between the original data and the result data, determine the original data and the result data located upstream of the result data of the data processing link as the upstream data of the result data of the data processing link; According to the data flow direction between the original data and the result data, and the data flow direction between the result data of the data processing link and the upstream data, the program indexes of the result data of the data processing link and the upstream data are sorted to determine the index chain; According to the index chain, the index information of the table node connection corresponding to the result data of the data processing link is determined.

6. The method according to claim 5, characterized in that The method further comprises: Using the program index of the data processing link in the index chain as the main information; Using the index chain as folding information of the main information; In response to a user's abnormal query command for the result data of the data processing link, determine a table node corresponding to the result data of the data processing link, and display the main information; In response to the user's command to expand the main information, the collapsed information of the main information is displayed.

7. The method according to claim 1, characterized in that The blood relationship link graph also includes column nodes; The method further comprises: For each table node in the blood relationship link graph, determine each column node connected to the table node according to each data column included in the original data or result data corresponding to the table node; For each data column included in the original data or result data corresponding to the table node, determine a program statement corresponding to the data column in the operation code of the data processing operation corresponding to the table node; Determine, according to the program statement corresponding to the data column, a data processing rule corresponding to the data column, and determine the data column used by the data processing rule as a source column; An edge from the source column to the data column is determined.

8. A data lineage determination device, characterized in that: include: A recording module determines a target business, and in each data processing link of the target business, records a program file corresponding to an operation code of a data processing operation of the data processing link, and records a position of the operation code in the program file; A program index determination module, which determines the program index of the data processing link according to the program file corresponding to the recorded operation code and the position of the operation code in the program file; A data flow direction determination module determines the original data of the target business and the result data obtained in each data processing link, and determines the data flow direction between the original data and each result data according to the operation code of the data processing operation of each data processing link; A construction module is provided to determine the nodes corresponding to the original data and the nodes corresponding to the result data as table nodes, determine the edges between the table nodes according to the data flow direction between the original data and the result data, and determine the index information of the table nodes corresponding to the result data of each data processing link according to the program index of each data processing link; The data lineage determination module determines a lineage link graph according to the table nodes, the edges between the table nodes and the index information of the table nodes.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images