Data consanguinity traceability analysis method based on power grid data center

Abstract

The invention provides a data consanguinity traceability analysis method based on a power grid data center. The method comprises the steps that an execution log is generated and collected based on Hive and YARN, a plurality of SQL statements are obtained based on the execution log, AST trees are constructed based on the SQL statements respectively, and a data consanguinity mapping relation and data node information are obtained based on the AST trees; and combining the same data nodes in the plurality of SQL statements based on an information entropy data alignment method, converting the data consanguinity mapping relation and the data nodes into a data consanguinity map based on Neo4j, and performing data path tracing on the data consanguinity map based on breadth-first search and an A * algorithm. Through the technical scheme of the invention, the occurrence position of problem data can be quickly and accurately positioned, the whole process of power grid data from acquisition, transmission, storage to analysis is transparent and traceable, and the requirements of enterprises for timeliness and reliability of data service in data are met.

Description

Technical Field

[0001] The present invention relates to the field of power grid data center data service, and in particular to a data lineage tracing and analysis method based on a power grid data center. Background Art

[0002] In recent years, the data middle platform has played an increasingly important role in a company's production and operations by providing centralized data management and processing capabilities, helping companies improve data quality, optimize decision-making, improve business efficiency, and support innovation while ensuring data security and compliance.

[0003] However, in supporting key front-end applications, the timeliness and reliability of the data services provided by the data center cannot meet the needs of enterprise data application products. The entire data link from data collection to data provision to front-end applications is extremely complex, involving aggregation, cleaning, storage, analysis and processing. The entire data link from the source to the application is a mesh structure, which brings huge challenges to the location of problematic data. Therefore, it is urgent to provide a solution to improve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide a data lineage tracing analysis method based on a power grid data center, which can quickly and accurately locate the location where the problematic data occurs.

[0005] The present invention provides a data lineage tracing and analysis method based on a power grid data center, which adopts the following technical solutions: Generate and collect execution logs based on Hive and YARN, obtain multiple SQL statements based on the execution logs, build AST trees based on the multiple SQL statements, and obtain data lineage mapping relationships and data node information based on the AST trees; Generate a data set by classifying based on data node information, calculate the influence of each attribute element in the data set on distinguishing data item by item based on the information entropy algorithm, select the attribute with the greatest influence as the key attribute of the data node, align the data nodes based on the value of the key attribute, and associate and merge the data nodes with the same key attribute value; The data lineage mapping relationship and data nodes are converted into a data lineage graph based on Neo4j, and the data path of the data lineage graph is traced based on breadth-first search and A* algorithm.

[0006] As described above, the data lineage tracing and analysis method based on the power grid data center, wherein the process of generating and collecting execution logs based on Hive and YARN includes: obtaining SQL statements submitted by users based on Hive, requesting resource allocation based on YARN and executing the SQL statements, and recording the information of executing the SQL statements in the execution logs.

[0007] The data lineage tracing analysis method based on the power grid data center as described above, wherein the process of constructing AST trees based on the multiple SQL statements respectively includes: The blank characters and comments of the SQL statement are removed and the statement is decomposed into SQL tokens, the tokens are identified and parsed based on a context-free grammar to obtain grammar rules, and an AST tree is constructed based on the grammar rules and a recursive descent parser.

[0008] As described above, the data lineage tracing analysis method based on the power grid data center station, wherein the process of constructing an AST tree based on grammatical rules and a recursive descent parser includes: A parsing function is designed based on a recursive descent parser, and whether the token conforms to the grammatical rules is determined based on the parsing function. When it conforms, a corresponding AST tree node is created, and the AST tree nodes are connected based on a recursive call sequence to obtain an AST tree, wherein the AST tree nodes include assignment nodes, function call nodes, and expression nodes.

[0009] The data lineage tracing analysis method based on the power grid data center as described above, wherein the process of obtaining the data lineage mapping relationship and data node information based on the AST tree includes: Initialize an empty dictionary for storing information, traverse the AST tree based on a recursive function and return it to the dictionary, record the mapping relationship between variables and assignment expressions in the assignment node during the traversal of the AST tree, record the function name and parameters in the function call node, and the function call and parameter dependency, recursively process the expression node, and obtain the internal data lineage mapping relationship and node information.

[0010] In the above-mentioned data lineage tracing analysis method based on the power grid data center, in the process of calculating the influence of each attribute element in the data set on distinguishing data item by item based on the information entropy algorithm, the following formula is used for calculation: ; in, Representing a collection of data The total number of different SQL statements or categories in express Middle The probability of an event or category is expressed as Indicates The logarithmic probability of an event, Representing a collection of data Information entropy.

[0011] As described above, the data lineage tracing and analysis method based on the power grid data center, wherein the process of tracing the data path of the data lineage graph based on breadth-first search and A* algorithm includes: determining the target node based on the data lineage graph, performing table-level lineage analysis based on breadth-first search traversal, performing field-level lineage analysis based on A* algorithm, and generating path information for the target node lineage analysis.

[0012] In the above-mentioned data lineage tracing analysis method based on the power grid data center, the process of performing table-level lineage analysis based on breadth-first search traversal includes: A table-level graph is constructed based on a data lineage graph, wherein nodes in the data lineage graph represent data tables, and directed edges in the data lineage graph represent dependencies between the data tables; a node is randomly selected as a traversal starting point, a BFS queue is initialized, the starting point is added to the queue and marked as visited, a node is taken out of the queue and all its adjacent nodes are checked, and unvisited adjacent nodes are added to the queue; during the traversal process, the lineage information of each node is recorded, and the relationship between the nodes is updated, and the breadth-first search is continued until the queue is empty and the traversal is completed; based on the collected lineage information, a table-level lineage relationship report is generated.

[0013] The data lineage tracing analysis method based on the power grid data center as described above, wherein the process of performing field-level lineage analysis based on the A* algorithm includes: Construct the field level based on the data lineage graph, the nodes in the data lineage graph represent data fields, and the directed edges in the data lineage graph represent the relationship between fields; determine the target field for analysis, and define a heuristic function to estimate the optimal path cost from the current field to the target field, set the starting field as the initial node of the A* algorithm, initialize the open list and the closed list, select the node with the lowest total cost from the open list for expansion, explore all adjacent nodes, and update the open list; during the traversal process, record the lineage information of each field, and update the relationship between nodes, and continue to execute the A* algorithm until the target field is expanded and reached, or the open list is empty and the traversal is completed; generate a field-level lineage relationship report based on the collected field-level lineage information.

[0014] The data lineage tracing analysis method based on the power grid data center proposed in the present invention has the following beneficial effects: 1. The present invention associates and merges data nodes with the same key attribute values ​​through information entropy calculation, ensuring that similar data nodes are effectively integrated, thereby improving the accuracy and comprehensiveness of data lineage results.

[0015] 2. The data lineage diagram of the present invention records the entire process of data from its original source to its final use, including the generation, transformation and flow path of data.

[0016] 3. The present invention can quickly and accurately locate the location where the problematic data occurs by generating a table-level lineage relationship report and a field-level lineage relationship report. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flow chart of a data lineage tracing and analysis method based on a power grid data center provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0019] The embodiment of the present invention provides a data lineage tracing analysis method based on a power grid data center. Figure 1 ,include: S1. Generate and collect execution logs based on Hive and YARN, obtain multiple SQL statements based on the execution logs, build AST trees based on the multiple SQL statements, and obtain data lineage mapping relationships and data node information based on the AST trees; S2. Classify and generate a data set based on the data node information, calculate the influence of each attribute element in the data set on distinguishing data item by item based on the information entropy algorithm, select the attribute with the greatest influence as the key attribute of the data node, align the data nodes based on the value of the key attribute, and associate and merge the data nodes with the same key attribute value; S3. Merge the same data nodes in the multiple SQL statements based on the information entropy data alignment method, convert the data lineage mapping relationship and data nodes into a data lineage graph based on Neo4j, and trace the data path of the data lineage graph based on breadth-first search and A* algorithm.

[0020] In some embodiments, the process of executing step S1 includes: S1.1. Generate and collect execution logs based on Hive and YARN, and obtain multiple SQL statements based on the execution logs; S1.2, constructing AST trees based on the multiple SQL statements respectively; S1.3. Obtain data lineage mapping relationship and data node information based on the AST tree.

[0021] Specifically, when executing step S1.1, generating and collecting execution logs based on Hive and YARN, and obtaining multiple SQL statements based on the execution logs, it includes: The SQL statement submitted by the user is obtained based on Hive, resource allocation is requested based on YARN, and the SQL statement is executed, and the information of executing the SQL statement is recorded in the execution log.

[0022] In fact, it is necessary to set the log level in the Hive configuration file hive-site.xml and ensure that the execution logging function is enabled. At this time, the user submits the SQL query through the Hive CLI, Hive submits the query task to the YARNResourceManager and requests the required allocated resources. YARN starts an ApplicationMaster to manage the execution of the SQL query; the ApplicationMaster assigns the task to multiple containers for parallel execution. When each container executes the task, it writes the log to HDFS. The log aggregation function of YARN will centralize the logs of these containers in HDFS for subsequent viewing and analysis.

[0023] Furthermore, the execution log is integrated into the log management tool ELK; the ELK stack consists of three main components: Logstash, Elasticsearch, and Kibana. Logstash is responsible for collecting log files from Hive and YARN and preprocessing log data. Its configuration includes input plug-ins, filters, and output plug-ins. The input plug-in is used to read data from log files, the filter is used to clean and convert data, and the output plug-in sends the processed data to Elasticsearch. Elasticsearch is a distributed search and analysis engine that is responsible for storing log data processed by Logstash and provides efficient query and analysis capabilities. Kibana is used to create dashboards and visual reports, and monitors the log information of Hive and YARN in real time by visually displaying the data in Elasticsearch. In addition, Kibana is configured with monitoring and alarm functions, which can detect abnormal information in logs in real time and trigger alarm mechanisms to respond to system failures in a timely manner.

[0024] Specifically, when executing step S1.2, respectively constructing AST trees based on the multiple SQL statements, it includes: S1.2.1, remove blank characters and comments from the SQL statement and decompose it into SQL tokens; S1.2.2, identifying and parsing the tokens based on a context-free grammar to obtain grammatical rules; S1.2.3. Build an AST tree based on grammar rules and recursive descent parser.

[0025] Specifically, when executing step S1.2.1, the blank characters and comments of the SQL statement are removed and decomposed into SQL tags, the blank characters include spaces, tabs and line breaks; the tags represent a basic element in the SQL statement, including keywords, identifiers, operators, values, etc.

[0026] Furthermore, when executing step S1.2.2, the tokens are identified and parsed based on the context-free grammar to obtain grammatical rules, the component structures of the tokens in the SQL statement are identified, and the tokens are classified to obtain grammatical rules.

[0027] Specifically, when executing step S1.2.3, building an AST tree based on grammar rules and a recursive descent parser, it includes: designing a parsing function based on the recursive descent parser, judging whether the tag complies with the grammar rules based on the parsing function, creating corresponding AST tree nodes when they comply, and connecting the AST tree nodes based on the recursive call sequence to obtain the AST tree, wherein the AST tree nodes include assignment nodes, function call nodes, and expression nodes.

[0028] Specifically, when executing step S1.3, obtaining the data lineage mapping relationship and data node information based on the AST tree includes: Initialize an empty dictionary for storing information, traverse the AST tree based on a recursive function and return it to the dictionary, record the mapping relationship between variables and assignment expressions in the assignment node during the traversal of the AST tree, record the function name and parameters in the function call node, and the function call and parameter dependency, recursively process the expression node, and obtain the internal data lineage mapping relationship and node information.

[0029] In some embodiments, the process of executing step S2 includes: S2.1, classify and generate data sets based on data node information; S2.2, calculating the influence of each attribute element in the data set on distinguishing data item by item based on the information entropy algorithm; S2.3. Align the data nodes based on the values ​​of key attributes.

[0030] Specifically, when executing step S2.1, classifying and generating a data set based on data node information includes: creating a relationship table for storing data lineage mapping relationships and node information and preparing a parsing tool, and classifying and generating a data set based on the names of the data nodes.

[0031] Specifically, when executing step S2.2, calculating the influence of each attribute element in the data set on distinguishing data item by item based on the information entropy algorithm, it includes: calculating the influence of each attribute element in the data set on distinguishing data item by item based on the information entropy algorithm, and selecting the attribute with the greatest influence as the key attribute of the data node.

[0032] Furthermore, in the process of calculating the influence of each attribute element in the data set on distinguishing data item by item based on the information entropy algorithm, the following formula is used for calculation: ; in, Representing a collection of data The total number of different SQL statements or categories in express Middle The probability of an event or category is expressed as Indicates The logarithmic probability of an event, Representing a collection of data Information entropy.

[0033] Furthermore, based on the above information entropy, the evaluation attribute can be obtained right The calculation formula of the influence degree is: ; in, for Medium Attributes The value set of Representing a collection of data Medium Attributes The value is A subset of the data, express Medium Attributes The value is The proportion of the data subset, Representation attributes The value is The information entropy of a subset of For attributes right degree of impact.

[0034] In fact, the key attributes have the most significant distinguishing ability and can effectively distinguish By associating and merging the data nodes with the same key attribute values, the accuracy and comprehensiveness of the data lineage results can be improved.

[0035] Specifically, when executing step S2.3, aligning the data nodes based on the values ​​of key attributes includes: associating and merging the data nodes with the same key attribute values ​​based on the values ​​of the key attributes, and inserting the processed data lineage mapping relationship and node information into a preset relationship table.

[0036] In some embodiments, the process of executing step S3 includes: S3.1. Convert the data lineage mapping relationship and data nodes into a data lineage graph based on Neo4j; S3.2. Tracing the data path of the data lineage graph based on breadth-first search and A* algorithm.

[0037] Specifically, when executing step S3.1, the data lineage mapping relationship and data nodes are converted into a data lineage graph based on Neo4j, the data lineage mapping relationship and node information in the relationship table are converted into a data lineage graph containing directed edges based on Neo4j.

[0038] In fact, the data lineage records the entire process from the original source to the final use of the data, including the generation, transformation and flow path of the data; traditional lineage tracking methods often rely on static reports and complex documents, which limits the dynamic display and analysis capabilities of data lineage information; this paper uses the graph database Neo4j to store and display data lineage in the form of a directed graph to improve the visualization and analysis capabilities of data lineage information.

[0039] Specifically, when executing step S3.2, tracing the data path of the data lineage graph based on breadth-first search and A* algorithm includes: S3.2.1, perform table-level kinship analysis based on breadth-first search traversal; S3.2.2. Perform field-level lineage analysis based on the A* algorithm.

[0040] Specifically, when executing step S3.2.1, performing table-level kinship analysis based on breadth-first search traversal includes: Constructing a table-level graph based on a data lineage graph, wherein nodes in the data lineage graph represent data tables, and directed edges in the data lineage graph represent dependency relationships between the data tables; Randomly select a node as the starting point of the traversal, initialize a BFS queue, add the starting point to the queue and mark it as visited, take a node from the queue and check all its adjacent nodes, and add the unvisited adjacent nodes to the queue; during the traversal process, record the lineage information of each node, and update the relationship between the nodes, and continue to perform breadth-first search until the queue is empty and the traversal is completed; based on the collected lineage information, generate a table-level lineage relationship report.

[0041] Specifically, when executing step S3.2.2, field-level lineage analysis based on the A* algorithm is performed, it includes: Constructing a field level based on a data lineage graph, wherein nodes in the data lineage graph represent data fields, and directed edges in the data lineage graph represent relationships between fields; Determine the target field for analysis and define a heuristic function to estimate the optimal path cost from the current field to the target field. Set the starting field as the initial node of the A* algorithm, initialize the open list and closed list, select the node with the lowest total cost from the open list for expansion, explore all adjacent nodes, and update the open list. During the traversal process, record the lineage information of each field and update the relationship between the nodes. Continue to execute the A* algorithm until the target field is expanded and reached, or the open list is empty and the traversal is completed. Generate a field-level lineage relationship report based on the collected field-level lineage information.

[0042] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. A data lineage tracing and analysis method based on a power grid data center, characterized in that: The following steps are involved: Generate and collect execution logs based on Hive and YARN, obtain multiple SQL statements based on the execution logs, build AST trees based on the multiple SQL statements, and obtain data lineage mapping relationships and data node information based on the AST trees; Classify and generate a data set based on data node information, calculate the influence of each attribute element in the data set on distinguishing data item by item based on the information entropy algorithm, select the attribute with the greatest influence as the key attribute of the data node, align the data nodes based on the value of the key attribute, and associate and merge the data nodes with the same key attribute value; The data lineage mapping relationship and data nodes are converted into a data lineage graph based on Neo4j, and data path tracing is performed on the data lineage graph based on breadth-first search and A* algorithm.

2. The data lineage tracing and analysis method based on the power grid data center according to claim 1 is characterized in that: The process of generating and collecting execution logs based on Hive and YARN includes: Obtain the SQL statement submitted by the user based on Hive, request resource allocation based on YARN and execute the SQL statement, and record the information of executing the SQL statement in the execution log.

3. The data lineage tracing and analysis method based on the power grid data center according to claim 1 is characterized in that: The process of constructing AST trees based on the multiple SQL statements includes: The SQL statement is stripped of whitespace characters and comments and decomposed into SQL tokens, the tokens are identified and parsed based on a context-free grammar to obtain grammar rules, and an AST tree is constructed based on the grammar rules and a recursive descent parser.

4. The data lineage tracing and analysis method based on the power grid data center according to claim 3 is characterized in that: The process of building an AST tree based on grammar rules and recursive descent parser includes: A parsing function is designed based on a recursive descent parser, and whether the token conforms to the grammatical rules is determined based on the parsing function. When it conforms, a corresponding AST tree node is created, and the AST tree nodes are connected based on the recursive call sequence to obtain an AST tree, where the AST tree nodes include assignment nodes, function call nodes, and expression nodes.

5. The data lineage tracing and analysis method based on the power grid data middle station according to claim 4 is characterized in that: The process of obtaining data lineage mapping relationships and data node information based on the AST tree includes: Initialize an empty dictionary for storing information, traverse the AST tree based on a recursive function and return it to the dictionary. During the AST tree traversal, the mapping relationship between variables and assignment expressions in the assignment node is recorded, the function name and parameters in the function call node, as well as the function call and parameter dependency, recursively process the expression node, and obtain the internal data lineage mapping relationship and node information.

6. The data lineage tracing and analysis method based on the power grid data center according to claim 1 is characterized in that: In the process of calculating the influence of each attribute element in the data set on distinguishing data item by item based on the information entropy algorithm, the following formula is used for calculation: ； in, Representing a data set The total number of different SQL statements or categories in express Middle The probability of an event or category is expressed as follows: Indicates the The logarithmic probability of an event, Representing a data set Information entropy.

7. The data lineage tracing and analysis method based on the power grid data center according to claim 1 is characterized in that: The process of tracing the data path of the data lineage graph based on breadth-first search and A* algorithm includes: The target node is determined based on the data lineage graph, table-level lineage analysis is performed based on breadth-first search traversal, field-level lineage analysis is performed based on the A* algorithm, and path information for the target node lineage analysis is generated.

8. The data lineage tracing and analysis method based on the power grid data center according to claim 7 is characterized in that: The process of table-level lineage analysis based on breadth-first search traversal includes: Constructing a table-level graph based on a data lineage graph, wherein the nodes in the data lineage graph represent data tables, and the directed edges in the data lineage graph represent dependency relationships between the data tables; A node is randomly selected as the starting point of the traversal, a BFS queue is initialized, the starting point is added to the queue and marked as visited, a node is taken from the queue and all its adjacent nodes are checked, and unvisited adjacent nodes are added to the queue; during the traversal process, the lineage information of each node is recorded and the relationship between the nodes is updated, and the breadth-first search is continued until the queue is empty and the traversal is completed; based on the collected lineage information, a table-level lineage relationship report is generated.

9. The data lineage tracing and analysis method based on the power grid data middle station according to claim 7 is characterized in that: The process of field-level lineage analysis based on the A* algorithm includes: Constructing a field level based on a data lineage graph, wherein nodes in the data lineage graph represent data fields, and directed edges in the data lineage graph represent relationships between fields; Determine the target field for analysis and define a heuristic function to estimate the optimal path cost from the current field to the target field. Set the starting field as the initial node of the A* algorithm, initialize the open list and closed list, select the node with the lowest total cost from the open list for expansion, explore all adjacent nodes, and update the open list. During the traversal process, record the lineage information of each field and update the relationship between the nodes. Continue to execute the A* algorithm until the target field is expanded and reached, or the open list is empty and the traversal is completed. Generate a field-level lineage relationship report based on the collected field-level lineage information.

Images