Data platform monitoring method, system and equipment based on data consanguinity and medium

Abstract

The invention provides a data platform monitoring method, system and device based on data consanguinity and a medium, and the method comprises the steps: regularly monitoring the execution state of a scheduling task on a data platform and the consistency of a data table, and when the scheduling task in an abnormal state or the inconsistent data table is monitored, sending the scheduling task to a server; and querying the data blood relationship graph to locate a problem source, determining an affected downstream scheduling task and data table and an abnormal front-end report page, and finally realizing end-to-end monitoring alarm. According to the invention, by adopting an automatic monitoring and alarming mechanism, the data processing efficiency is remarkably improved. Meanwhile, a data consanguinity graph is adopted to record data entities and associations thereof. A user can accurately position the source of an abnormal condition and the influence range thereof, and an important basis is provided for taking corrective measures in time. In addition, the method supports end-to-end monitoring alarm, is suitable for most big data development platforms in the market, is high in universality, and is economical and practical.

Description

Technical Field

[0001] The present application belongs to the field of big data technology and relates to a data platform monitoring method, system, device and medium based on data lineage. Background Art

[0002] When processing massive amounts of data, big data platforms usually involve complex data processing processes and diverse business logic. An effective monitoring mechanism can not only ensure the stability and reliability of the data platform, but also promptly detect and resolve performance bottlenecks, failures, and abnormal situations. In addition, monitoring can also help optimize resource allocation, reduce operating costs, and improve user experience. This is crucial to maintaining data integrity, improving decision-making efficiency and quality, and meeting regulatory compliance requirements.

[0003] The current big data platform monitoring method is mainly based on stream data processing technology, which realizes data reporting and storage by capturing the log service data of the big data platform. Subsequently, the functions of data display and platform monitoring are realized by calculating and analyzing these log service data. Although this solution can meet the needs to a certain extent, it also has some limitations. For example, when the network scale is large, the time required to locate abnormal problems will increase significantly, resulting in low efficiency and accuracy. Summary of the invention

[0004] The purpose of this application is to provide a data platform monitoring method, system, device and medium based on data lineage, which are used to solve the limitations of existing big data platform monitoring methods in terms of real-time and accuracy.

[0005] In a first aspect, the present application provides a data platform monitoring method based on data lineage, comprising: regularly monitoring the execution status of scheduling tasks on a data platform; the scheduling tasks include entry-to-lake tasks, ETL development tasks and data synchronization tasks; after the entry-to-lake tasks and the data synchronization tasks are completed, regularly checking the consistency of data tables; when an abnormal scheduling task is monitored or an inconsistent data table is detected, querying a pre-constructed data lineage relationship map; the data lineage relationship map comprises multiple lineage links, each of which comprises at least two nodes, and a relationship edge is provided between every two adjacent nodes; each of the nodes is used to represent a data table, and each of the relationship edges is used to represent a scheduling task; according to the upstream and downstream dependencies in the data lineage relationship map, locating the downstream affected objects of the abnormal scheduling task or the inconsistent data table; the downstream affected objects include downstream scheduling tasks, downstream data tables and front-end report pages.

[0006] In an implementation of the first aspect, the method for constructing the data blood relationship map includes:

[0007] Get all SQL statements executed in the data warehouse within a preset time period;

[0008] Parsing each of the SQL statements to generate a corresponding abstract syntax tree;

[0009] Parsing the abstract syntax tree to obtain lineage information contained in different nodes in the abstract syntax tree; the lineage information includes information of a source table, information of a target table, and a data flow path from the source table to the target table;

[0010] The source table is used as a data inflow node, the target table is used as a data outflow node, and the data flow path from the source table to the target table is used as a relationship edge to construct a bloodline link in the data bloodline relationship graph.

[0011] In an implementation of the first aspect, after the downstream affected object is located, an early warning mechanism is activated; the activation of the early warning mechanism includes:

[0012] Generate a formatted warning notification text based on the scheduling task in the abnormal state or the inconsistent data table;

[0013] A specific type of alarm interface is called, and the formatted alarm notification text is pushed to a relevant person in charge through the specific type of alarm interface.

[0014] In an implementation of the first aspect, a graph structure database neo4j is used to store the data blood relationship map.

[0015] In an implementation of the first aspect, querying a pre-built data lineage relationship graph includes: using the Cypher query language to deeply traverse the graph structure database neo4j to obtain upstream and downstream dependencies in the data lineage relationship graph.

[0016] In an implementation of the first aspect, after the lake entry task is completed, periodically checking the consistency of the data table includes:

[0017] Obtaining the timing data statistics results of the source end business system; the timing data statistics results are recorded in a data table of a specific mode under the source layer;

[0018] Execute SQL query on the data table at the source layer to obtain the actual number of records in the data table;

[0019] Compare the timing data statistics result with the actual number of records in the data table;

[0020] If the two are inconsistent, the difference between them will be recorded in the monitoring result table.

[0021] In an implementation of the first aspect, after the data synchronization task is completed, periodically checking the consistency of the data table includes:

[0022] Regularly obtain the table names of all data tables to be tested in the configuration table;

[0023] Querying the amount of data in each of the data tables at a data reporting layer or a data mart layer to obtain a first data query result;

[0024] Query the amount of data in the corresponding data table in the Doris data warehouse to obtain a second data query result; the data in the Doris data warehouse is obtained by synchronizing the result data of the data reporting layer or the data mart layer;

[0025] comparing the first data query result with the second data query result;

[0026] If the two are inconsistent, the difference between them will be recorded in the monitoring result table.

[0027] In the second aspect, the present application provides a data platform monitoring system based on data lineage, including: a task status monitoring module, which is used to periodically monitor the execution status of scheduling tasks on the data platform; the scheduling tasks include entry-to-lake tasks, ETL development tasks and data synchronization tasks; a consistency verification module, which is used to periodically verify the consistency of data tables after the entry-to-lake tasks and the data synchronization tasks are completed; a data lineage query module, which is used to query a pre-constructed data lineage relationship map when an abnormal scheduling task is monitored or an inconsistent data table is detected; the data lineage relationship map includes multiple lineage links, each of which includes at least two nodes, and a relationship edge is provided between every two adjacent nodes; each of the nodes is used to represent a data table, and each of the relationship edges is used to represent a scheduling task; an abnormality locating module, which is used to locate the downstream affected objects of the scheduling task in the abnormal state or the inconsistent data table according to the upstream and downstream dependencies in the data lineage relationship map; the downstream affected objects include downstream scheduling tasks, downstream data tables and front-end report pages.

[0028] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device performs any of the methods described above.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements any of the methods described above when executed by a processor.

[0030] As described above, the data platform monitoring method, system, device and medium based on data lineage described in this application have the following beneficial effects:

[0031] (1) The use of automated monitoring and alarm mechanisms reduces manual intervention and improves data processing efficiency.

[0032] (2) By using program code to monitor the execution status of scheduled tasks and the consistency of data tables on the data platform in real time, abnormal problems can be discovered accurately and promptly, thereby ensuring the stable operation of the data platform.

[0033] (3) A clear data lineage relationship map is used to record data entities and their relationships, which enhances the transparency of data management on the big data platform. Users can easily grasp the mutual influence between data, thereby quickly understanding the degree of relationship between data entities and improving the actual application value of data.

[0034] (4) By deeply traversing the blood relationship map, users can quickly and comprehensively track abnormal situations and accurately locate the source of the problem and its scope of impact, providing an important basis for taking corrective measures in a timely manner.

[0035] (5) It supports end-to-end monitoring and alarming, and is applicable to most big data development platforms on the market. It is highly versatile and economical. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Shown is a hardware structure block diagram of a backend management server in one embodiment of the present application.

[0037] Figure 2 Shown is a structural block diagram of a data warehouse architecture of the big data platform described in an embodiment of the present application.

[0038] Figure 3 Shown is a flow chart of an embodiment of the data platform monitoring method based on data lineage described in the embodiments of the present application.

[0039] Figure 4 Shown is a flow chart of another embodiment of the data platform monitoring method based on data lineage described in the embodiments of the present application.

[0040] Figure 5 Shown is a flow chart of yet another embodiment of the data platform monitoring method based on data lineage described in the embodiments of the present application.

[0041] Figure 6 Shown is a schematic diagram of a partial bloodline link in one embodiment of the data bloodline relationship map described in the embodiments of the present application.

[0042] Figure 7Shown is a schematic diagram of an abnormal front-end report page described in an embodiment of the present application in one embodiment.

[0043] Figure 8 Shown is a structural schematic diagram of an embodiment of the formatted notification text content described in the embodiments of the present application.

[0044] Fig. 9 Shown is a structural diagram of a data platform monitoring system based on data lineage in one embodiment of the present application.

[0045] Fig.10 Shown is a schematic structural diagram of an electronic device described in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0047] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0048] In addition, in this application, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0049] The following embodiments of the present application provide a data platform monitoring method, system, device and medium based on data lineage, which can help enterprises achieve data governance of big data platforms and ensure data compliance and consistency. By monitoring data flow, data anomalies can be discovered and processed in a timely manner, thereby providing accurate data support for business decisions, thereby improving the efficiency and quality of decision-making.

[0050] The data platform monitoring method based on data lineage provided in the embodiment of the present application can be run in the backend management server. Figure 1 , which is a hardware structure diagram of a backend management server in an embodiment of the present application. Figure 1 As shown, the server includes: a processor and a memory, the processor may be a central processing unit, and the memory is used to store data. Figure 1 The server in the example is only used for example and does not limit the specific structure of the server.

[0051] Optionally, the server may further include: communication transmission equipment and input and output equipment.

[0052] Optionally, the memory may be used to store computer programs, such as software programs and modules of application software, and the memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0053] Optionally, the communication transmission device can be used to receive or send data via a network, which may include a wireless network provided by a communication provider. The communication transmission device may include a NIC (Network Interface Controller) that can be connected to other network devices through a base station so as to communicate with the Internet.

[0054] Optionally, the input and output device may include an intuitive user interface, allowing management personnel to easily view monitoring data and alarm information.

[0055] See also Figure 2 , which is a structural block diagram of the data warehouse architecture of the big data platform described in the embodiment of the present application in one embodiment. Figure 2 As shown, the data warehouse architecture includes a source layer, a data integration layer (Data Warehouse Integration, DWI), a data report layer (Data Warehouse Report, DWR), a data mart layer (Data Mart, DM), Doris data warehouse and report application.

[0056] Specifically, as the starting point of the data warehouse architecture, the source layer is directly connected to the source business system and uses advanced technologies such as Hadoop Upstream Data Infrastructure (HUDI) to store structured, semi-structured and unstructured data from different source business systems. This process achieves the initial landing of source business system data and ensures the integrity and availability of the data.

[0057] The data integration layer is responsible for efficiently integrating and deeply cleaning data from different source business systems. At this layer, data will be converted into a unified format, data redundancy will be eliminated, missing values ​​will be filled, and necessary data conversion will be performed. The integrated data will be more standardized and consistent, ensuring data quality and consistency.

[0058] The data reporting layer and the data integration layer are consistent in data granularity to ensure the consistency and accuracy of information. This layer focuses on generating detailed reports and intuitive dashboards to provide users with a clear display of data and help users deeply understand the meaning of the data and the trends it reveals.

[0059] The data mart layer focuses on meeting the data needs of specific business areas. At this layer, data is further refined and optimized to accurately support specific analysis needs and decision-making processes, providing strong data support for the formulation of business strategies.

[0060] The data results generated by the data reporting layer and the data mart layer are usually synchronized to the Doris data warehouse for data display on the front-end report page. The Doris data warehouse is a high-performance analytical database with high concurrent query processing capabilities and fast data scanning performance. It can effectively support real-time analysis and visualization of large-scale data, providing strong data support for decision-making.

[0061] The reporting application provides an interface for users to interact with data, allowing users to conduct in-depth data analysis and make wise decisions by viewing data reports, thereby improving the accessibility and practicality of data.

[0062] The entire data warehouse architecture realizes efficient data flow and management from the original data of the source business system to the final report display. This end-to-end data transmission form ensures the integrity, consistency and availability of data throughout the entire process, providing reliable data support for business decision-making.

[0063] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0064] See also Figure 3, which is a flow chart of an embodiment of the data platform monitoring method based on data lineage described in the embodiment of the present application. Figure 3 As shown, an embodiment of the present application provides a data platform monitoring method based on data lineage, including the following steps S100 to S400.

[0065] In step S100, the execution status of the scheduled tasks on the data platform is regularly monitored.

[0066] In one embodiment of the present application, the scheduling tasks on the big data platform are mainly divided into the following categories:

[0067] (1) Lake entry task: mainly responsible for extracting data from the source business system and storing it in the source layer.

[0068] (2) ETL development tasks: Based on the data in the source layer, a series of complex data integration operations such as extraction, transformation, and loading are performed. Subsequently, the processed data is transferred to the data reporting layer for further processing to meet specific analysis requirements. Finally, the data is pushed to the data mart layer to support more refined business decision making.

[0069] (3) Data synchronization task: Mainly responsible for synchronizing the data results generated by the data reporting layer and the data mart layer to the Doris data warehouse in real time for data display on the front-end report page.

[0070] In one embodiment of the present application, the execution status of the scheduled task includes: task submission successful, task in progress, task about to be suspended, task already suspended, task about to be stopped, task already stopped, task execution failed, task execution successful, task requires fault tolerance, task forced to terminate, task waiting for available thread resources, task waiting for dependent tasks to complete, task forced to succeed, and task being dispatched. In theory, except for the state of "task execution successful", other execution states are considered abnormal states.

[0071] In order to ensure timely monitoring of the execution status of the scheduling task, the backend management server can be used to deploy the scheduled task on the big data platform. In the embodiment of the present application, the design of the scheduled task fully considers the actual business needs and allows the execution frequency to be flexibly set, such as automatically executing once every hour or every day, thereby ensuring continuous tracking and instant feedback of the running status of the scheduling task, providing strong support for the stable operation of the system and timely fault response.

[0072] Data related to the scheduling task, such as key information such as the running time of the scheduling task, is usually stored in the underlying database. In the embodiment of the present application, MySQL is selected as the underlying database to support the storage and management of data with its stability and efficiency. At the same time, a Python script is written in the corresponding underlying database, and the Python script is executed based on the scheduled task configured by the back-end management server, thereby realizing the automatic monitoring and acquisition of the scheduling task status. Once the Python script monitors the scheduling task in an abnormal state, the relevant information will be written into the monitoring result table.

[0073] In one embodiment of the present application, the Python script may be a view program. The view program has a data visualization function. Specifically, the Python script may use data visualization libraries such as matplotlib, seaborn, and plotly to display the execution status, running time, and other performance indicators of the scheduling task in the form of a chart, making the data more intuitive and easy to understand.

[0074] In addition, the view program can create an interactive user interface by integrating graphical user interface (GUI) libraries such as Tkinter and PyQt. This interface allows users to directly view the current status of tasks, adjust scheduling parameters, or trigger the execution of tasks on the interface, thereby enhancing the user's operating experience and the interactivity of the system.

[0075] In step S200, after the lake entry task and the data synchronization task are completed, the consistency of the data table is checked regularly.

[0076] In order to ensure that the consistency of the data table is checked in time, the data timing statistics task of the source business system can be deployed on the big data platform using the back-end management server. In the embodiment of the present application, the design of the data timing statistics task fully considers the actual business needs and allows the execution frequency to be flexibly set, such as automatically executing once at 12:00 every day and automatically executing once at 17:00 in the afternoon, so as to continuously track and instantly feedback the consistency of the data table, and provide strong support for the stable operation of the system and timely fault response.

[0077] The embodiment of the present application can also write a Python script and execute the script based on the data timing statistics task, so as to realize the automatic comparison and verification of the consistency of the data table. Once the Python script monitors the inconsistent data table, the relevant information will be written into the monitoring result table.

[0078] See also Figure 4 , which is a flow chart of another embodiment of the data platform monitoring method based on data lineage described in the embodiment of the present application. Figure 4As shown, in one embodiment of the present application, after the lake entry task is completed, the consistency of the data table is regularly checked, including the following steps S201 to S204.

[0079] In step S201, the timing data statistics result of the source business system is obtained.

[0080] Specifically, the timing data statistics are recorded in a data table of a specific mode under the source layer.

[0081] In step S202, an SQL query is executed on the data table in the source layer to obtain the actual number of records in the data table.

[0082] In step S203, the timing data statistics result is compared with the actual number of records in the data table.

[0083] In step S204, if the two are inconsistent, the difference between the two is recorded in the monitoring result table.

[0084] The core goal of the lake entry task is to ensure the consistency between the data table extracted from the source business system and the data table stored in the source layer. However, in the actual operation process, due to factors such as technical errors or data transmission problems, inconsistencies in the data table often occur. To address this problem, this application adopts a strategy of performing data table consistency verification after the lake entry task is completed to discover inconsistencies in the data table, thereby effectively ensuring the integrity and accuracy of the data.

[0085] See also Figure 5 , which is a flow chart of another embodiment of the data platform monitoring method based on data lineage described in the embodiment of the present application. Figure 5 As shown, after the data synchronization task is completed, the consistency of the data table is regularly checked, including the following steps S205 to S209.

[0086] In step S205, the table names of all data tables to be checked in the configuration table are obtained in a timely manner.

[0087] In step S206, the amount of data in each of the data tables is queried at the data reporting layer or the data mart layer to obtain a first data query result.

[0088] In step S207, the amount of data in the corresponding data table is queried in the Doris data warehouse to obtain a second data query result; the data in the Doris data warehouse is synchronized with the result data of the data reporting layer or the data mart layer.

[0089] In step S208, the first data query result is compared with the second data query result.

[0090] In step S209, if the two are inconsistent, the difference between the two is recorded in the monitoring result table.

[0091] In theory, the data before and after data synchronization should be consistent. However, in actual operation, due to factors such as network delays and system failures, the data in the Doris data warehouse may be inconsistent with the result data at the data reporting layer or data mart layer. To address this problem, this application adopts a strategy of performing data table consistency verification after the data synchronization task is completed to promptly detect data table inconsistencies, thereby effectively ensuring the integrity and accuracy of the data.

[0092] In this implementation, the execution status of the scheduled tasks and the consistency of the data table on the data platform are monitored in real time through program code, which can accurately and timely discover abnormal problems, thereby ensuring the stable operation of the data platform.

[0093] In step S300, when an abnormal scheduling task is monitored or an inconsistent data table is detected, a pre-constructed data lineage relationship map is queried.

[0094] In one embodiment of the present application, the abnormal status of the scheduled task includes successful task submission, task in progress, task about to be paused, task already paused, task about to be stopped, task already stopped, task execution failed, task requires fault-tolerant processing, task is forcibly terminated, task is waiting for available thread resources, task is waiting for dependent tasks to complete, task is forced to succeed, and task is being dispatched.

[0095] In an embodiment of the present application, the data lineage relationship map is used to represent data lineage relationships at the table level. Specifically, the data lineage relationship map includes multiple lineage links, each of which includes at least two nodes, and a relationship edge is provided between every two adjacent nodes; each of the nodes is used to represent a data table, and each of the relationship edges is used to represent a scheduling task. Each data table has independent attributes. If the data processing or analysis of a data table B depends on another data table A, there will be a directed edge from data table A to data table B in the data lineage relationship map, indicating that data table A is the upstream data source of data table B. When the upstream data table A changes, the data lineage relationship map can help quickly identify the affected downstream data table B, so as to adjust the relevant data processing tasks in time.

[0096] In this implementation, a clear data lineage relationship map is used to record data entities and their associations, which enhances the transparency of data management on the big data platform. Users can easily grasp the mutual influence between data, thereby quickly understanding the degree of relationship between data entities and improving the actual application value of data.

[0097] In one embodiment of the present application, the method for constructing the data lineage relationship map includes: obtaining all SQL statements executed in the data warehouse within a preset time period; parsing each of the SQL statements to generate a corresponding abstract syntax tree; parsing the abstract syntax tree to obtain the lineage information contained in different nodes in the abstract syntax tree; the lineage information includes information of the source table, information of the target table, and a data flow path from the source table to the target table; using the source table as a data inflow node, the target table as a data outflow node, and the data flow path from the source table to the target table as a relationship edge to construct a lineage link in the data lineage relationship map.

[0098] like Figure 6 As shown, data table 919->data table 890->data table 2937->data table 91 together constitute a lineage link. In this lineage link, data table 919, data table 890, data table 2937 and data table 91 may be located in different hierarchical structures respectively. For example, data table 91 may be located in the report application layer, while data table 2937 may be located in the data report layer or the data mart layer. In particular, the ID number "13210420484672" between data table 919 and data table 890 is used to identify a specific scheduling task. According to the attribute parameters of data table 91, the business domain corresponding to data table 91 is the sales domain, which is closely related to the "sales domain / sales mobile terminal / sales order AND sales return order / sales return order details" on the front-end report page. Due to space limitations, other lineage links will not be elaborated in detail here.

[0099] In one embodiment of the present application, a graph structure database neo4j is used to store the data blood relationship graph.

[0100] Neo4j is a relational database management system (NoSQL) designed specifically for processing graph data. It uses a graph model to efficiently store and manage data. In this graph model, nodes represent entities, and edges represent associations or relationships between entities. This model is very suitable for describing complex networks in the real world, such as social networks, recommendation systems, and knowledge graphs.

[0101] Neo4j supports the Cypher query language, which can perform path queries and complex join operations, making it easy to find data with a specific relationship path. In addition, Neo4j also supports real-time data analysis and high-performance concurrent access, making it very suitable for application scenarios that require frequent linked data queries.

[0102] In one embodiment of the present application, querying a pre-constructed data lineage relationship graph includes: using the Cypher query language to deeply traverse the graph structure database neo4j to obtain upstream and downstream dependencies in the data lineage relationship graph.

[0103] In this implementation, the graph structure database neo4j is used to store data blood relationship maps, which can effectively improve query efficiency, increase the speed and accuracy of problem locating, and reduce troubleshooting time.

[0104] In step S400, the scheduling task in the abnormal state or the downstream affected object of the inconsistent data table is located according to the upstream and downstream dependency relationship in the data lineage relationship map.

[0105] Specifically, when an abnormal scheduling task or inconsistent data table is monitored, the embodiment of the present application determines the affected downstream objects by querying the Neo4j database. The downstream affected objects include downstream scheduling tasks, downstream data tables, and front-end report pages.

[0106] See also Figure 7 , which is a schematic diagram of an abnormal front-end report page in an embodiment of the present application. Figure 7 As shown, due to the abnormal execution status of a scheduling task on the bloodline link or the inconsistency of the data table, there is no graphical data in the "Purchasing and Sales Trend Analysis Window" in the front-end report page.

[0107] In this implementation, users can quickly and comprehensively track abnormal situations by deeply traversing the blood relationship map, and accurately locate the source of the problem and its scope of impact, providing an important basis for taking corrective measures in a timely manner.

[0108] In one embodiment of the present application, the data platform monitoring method based on data lineage provided by the present application also includes step S500.

[0109] In step S500, after the affected downstream object is located, the early warning mechanism is activated.

[0110] In one embodiment of the present application, the activation of the early warning mechanism includes the following steps S501 to S502.

[0111] In step S501, a formatted alarm notification text is generated based on the abnormal state scheduling task or the inconsistent data table.

[0112] In step S502, a specific type of alarm interface is called, and the formatted alarm notification text is pushed to a relevant person in charge through the specific type of alarm interface.

[0113] In the embodiment of the present application, the formatted notification text can contain detailed abnormal information to ensure that the recipient can quickly understand the abnormal problem. The formatted notification text content significantly enhances the readability of the information. Figure 8 , showing a structural diagram of the formatted notification text content described in an embodiment of the present application in one embodiment.

[0114] It should be emphasized that the alarm interface in the embodiment of the present application is flexible and diverse, and can be in the form of text messages, phone calls, etc. The present application does not restrict the contact method, and aims to provide a variety of alarm solutions.

[0115] This implementation supports end-to-end monitoring and alarming, and is applicable to most big data development platforms on the market. It is highly versatile and economical.

[0116] It should be noted that the protection scope of the data platform monitoring method based on data lineage described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment, and all solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0117] See also Fig. 9 , showing a structural diagram of a data platform monitoring system based on data lineage in an embodiment of the present application.

[0118] like Fig. 9 As shown, an embodiment of the present application provides a data platform monitoring system based on data lineage, the system comprising a task status monitoring module, a consistency verification module, a data lineage query module and an anomaly location module.

[0119] In one embodiment of the present application, the task status monitoring module is used to periodically monitor the execution status of scheduled tasks on the data platform.

[0120] The scheduling tasks include lake entry tasks, ETL development tasks and data synchronization tasks.

[0121] The consistency check module is used to periodically check the consistency of the data table after the lake entry task and the data synchronization task are completed.

[0122] The data lineage query module is used to query a pre-constructed data lineage relationship map when an abnormal scheduling task is monitored or an inconsistent data table is detected; the data lineage relationship map includes multiple lineage links, each of which includes at least two nodes, and a relationship edge is provided between every two adjacent nodes; each of the nodes is used to represent a data table, and each of the relationship edges is used to represent a scheduling task.

[0123] The abnormality locating module is used to locate the downstream affected objects of the abnormal scheduling task or the inconsistent data table according to the upstream and downstream dependency relationship in the data lineage relationship map. The downstream affected objects include downstream scheduling tasks, downstream data tables and front-end report pages.

[0124] It should be noted that the structures and principles of the task status monitoring module, consistency verification module, data lineage query module and anomaly location module described in the embodiments of the present application correspond one-to-one to the steps in the above-mentioned data platform monitoring method based on data lineage, so they will not be repeated here.

[0125] The data platform monitoring system based on data lineage provided in the embodiment of the present application can implement the data platform monitoring method based on data lineage described in the present application, but the implementation device of the data platform monitoring method based on data lineage described in the present application includes but is not limited to the structure of the data platform monitoring system based on data lineage listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0126] See also Fig.10 , which is a schematic diagram of the structure of an electronic device described in an embodiment of the present application.

[0127] like Fig.10 As shown, an embodiment of the present application provides an electronic device, including: a processor and a memory.

[0128] The memory is used to store computer programs.

[0129] The processor is used to execute the computer program stored in the memory so that the electronic device executes any one of the above methods.

[0130] Preferably, the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0131] This embodiment also includes one or more of a multimedia component, an input / output (I / O) interface, and a communication component.

[0132] The multimedia component may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in a memory or sent through a communication component. The audio component also includes at least one speaker for outputting audio signals. The I / O interface provides an interface between the processor and other interface modules, and the other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component is used for wired or wireless communication between the timer and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component may include: a Wi-Fi module, a Bluetooth module, an NFC module.

[0133] In one embodiment, the timer can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned drone network deployment and collaborative caching methods.

[0134] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0135] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0136] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0137] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by the processor, the method described in any of the above items is implemented. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital videodisc, DVD)), or a semiconductor medium (e.g., a solid-state hard disk (solid state disk, SSD)), etc.

[0138] The present application embodiment may also provide a computer program product, the computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer or data center.

[0139] When the computer program product is executed by a computer, the computer executes the method described in the above method embodiment. The computer program product may be a software installation package, and when the above method is required, the computer program product may be downloaded and executed on a computer.

[0140] In summary, the present application provides a data platform monitoring method, system, device and medium based on data lineage, the method includes regularly monitoring the execution status of the scheduling task on the data platform and the consistency of the data table. When the abnormal state of the scheduling task or the inconsistent data table is monitored, the data lineage relationship map will be queried to locate the source of the problem, and the affected downstream scheduling tasks and data tables and the abnormal front-end report page will be determined, and the end-to-end monitoring alarm will be finally realized. The present application adopts an automated monitoring and alarm mechanism to reduce manual intervention and improve the efficiency of data processing; the execution status of the scheduling task on the data platform and the consistency of the data table are monitored in real time through the program code, and abnormal problems can be accurately and timely discovered, thereby ensuring the stable operation of the data platform. At the same time, a clear data lineage relationship map is used to record data entities and their associations, which enhances the transparency of data management on the big data platform. Users can easily grasp the mutual influence between data, so as to quickly understand the degree of relationship between data entities and improve the actual application value of data. By deeply traversing the lineage relationship map, users can quickly and comprehensively track abnormal situations, and accurately locate the source of the problem and its scope of influence, providing an important basis for taking corrective measures in a timely manner. In addition, this application supports end-to-end monitoring and alarming, and is suitable for most big data development platforms on the market. It is highly versatile and economical.

[0141] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0142] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A data platform monitoring method based on data lineage, characterized in that: include: Regularly monitor the execution status of scheduled tasks on the data platform; The scheduling tasks include lake entry tasks, ETL development tasks and data synchronization tasks; After the lake entry task and the data synchronization task are completed, the consistency of the data table is checked regularly; When an abnormal scheduling task is monitored or an inconsistent data table is detected, a pre-constructed data lineage relationship map is queried; the data lineage relationship map includes multiple lineage links, each lineage link includes at least two nodes, and a relationship edge is provided between every two adjacent nodes; each node is used to represent a data table, and each relationship edge is used to represent a scheduling task; According to the upstream and downstream dependencies in the data lineage relationship map, the downstream affected objects of the abnormal scheduling task or the inconsistent data table are located; the downstream affected objects include downstream scheduling tasks, downstream data tables and front-end report pages.

2. The method according to claim 1, characterized in that The method for constructing the data blood relationship map includes: Get all SQL statements executed in the data warehouse within a preset time period; Parsing each of the SQL statements to generate a corresponding abstract syntax tree; Parsing the abstract syntax tree to obtain lineage information contained in different nodes in the abstract syntax tree; the lineage information includes information of a source table, information of a target table, and a data flow path from the source table to the target table; The source table is used as a data inflow node, the target table is used as a data outflow node, and the data flow path from the source table to the target table is used as a relationship edge to construct the bloodline link in the data bloodline relationship graph.

3. The method according to claim 1, characterized in that It also includes activating early warning mechanisms after locating affected objects downstream; The startup early warning mechanism includes: Generate a formatted warning notification text based on the scheduling task in the abnormal state or the inconsistent data table; A specific type of alarm interface is called, and the formatted alarm notification text is pushed to a relevant person in charge through the specific type of alarm interface.

4. The method according to claim 1, characterized in that: The graph structure database neo4j is used to store the data blood relationship map.

5. The method according to claim 4, characterized in that Querying the pre-built data lineage relationship graph includes: using Cypher query language to deeply traverse the graph structure database neo4j to obtain upstream and downstream dependencies in the data lineage relationship graph.

6. The method according to claim 1, characterized in that After the lake entry task is completed, the consistency of the data table is regularly checked including: Obtaining the timing data statistics results of the source end business system; the timing data statistics results are recorded in a data table of a specific mode under the source layer; Execute SQL query on the data table at the source layer to obtain the actual number of records in the data table; Compare the timing data statistics result with the actual number of records in the data table; If the two are inconsistent, the difference between them will be recorded in the monitoring result table.

7. The method according to claim 1, characterized in that After the data synchronization task is completed, the consistency of the data table is checked regularly, including: Regularly obtain the table names of all data tables to be tested in the configuration table; Querying the amount of data in each of the data tables at a data reporting layer or a data mart layer to obtain a first data query result; Query the amount of data in the corresponding data table in the Doris data warehouse to obtain a second data query result; the data in the Doris data warehouse is obtained by synchronizing the result data of the data reporting layer or the data mart layer; comparing the first data query result with the second data query result; If the two are inconsistent, the difference between them will be recorded in the monitoring result table.

8. A data platform monitoring system based on data lineage, characterized in that: include: The task status monitoring module is used to periodically monitor the execution status of scheduled tasks on the data platform; The scheduling tasks include lake entry tasks, ETL development tasks and data synchronization tasks; A consistency check module, used to periodically check the consistency of the data table after the lake entry task and the data synchronization task are completed; A data lineage query module is used to query a pre-constructed data lineage relationship map when an abnormal scheduling task is monitored or an inconsistent data table is detected; the data lineage relationship map includes multiple lineage links, each of which includes at least two nodes, and a relationship edge is provided between every two adjacent nodes; each node is used to represent a data table, and each relationship edge is used to represent a scheduling task; An exception location module is used to locate the scheduling task in the abnormal state or the downstream affected object of the inconsistent data table according to the upstream and downstream dependencies in the data lineage relationship map; the downstream affected objects include downstream scheduling tasks, downstream data tables and front-end report pages.

9. An electronic device, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory so as to enable the electronic device to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images