Full-link data monitoring method and device

Abstract

The invention provides a full-link data monitoring method and device. Task information, collection program monitoring data, offline data warehouse program monitoring data and real-time data warehouse program monitoring data of a data governance task are obtained from a data governance platform, a virtual machine cluster and a kubernetes cluster; executing a first monitoring process for task information of the data governance task; and executing a second monitoring process for the acquisition program monitoring data, the offline counting program monitoring data and the real-time counting program monitoring data. Problems in a data link are identified through real-time anomaly detection. Quality closed-loop management is adopted, the integrity, consistency and accuracy of data are continuously monitored, a problem discovery, repair and verification closed loop is formed, and the data quality is gradually improved. Meanwhile, a complete link of data from a source to an application is tracked, a data source, processing logic and a use scene are clarified, and finally compliance, safety, quality and efficiency of the data in each link are ensured.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a full-link data monitoring method and device. Background Art

[0002] As enterprises undergo digital transformation, data volumes are increasing dramatically, and data types are becoming more diverse, including structured, unstructured, and semi-structured data. At the same time, enterprise storage environments are becoming increasingly complex, encompassing diverse deployment methods such as multi-cloud, hybrid cloud, and edge computing.

[0003] In this context, traditional manual management methods primarily focus on monitoring a single step, failing to fully manage the data lifecycle. Traditional data governance platforms are unable to provide end-to-end, real-time monitoring and analysis of data from generation to destruction. Furthermore, monitoring tools lack flexibility and are unable to adapt to complex and changing needs. Manual intervention and post-processing remain common practices, increasing costs and potentially leading to data leaks, quality degradation, and compliance risks.

[0004] Therefore, how to achieve efficient monitoring and control of the entire life cycle of data is an urgent problem that needs to be solved. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a full-link data monitoring method and device to solve the problem of comprehensive management and control of data life cycle.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of the present invention discloses a full-link data monitoring method, the method comprising:

[0008] Obtain task information for data governance tasks, collection program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the data governance platform, virtual machine cluster, and Kubernetes cluster.

[0009] For the task information of the data governance task, executing a first monitoring process;

[0010] executing a second monitoring process for the acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data;

[0011] Output monitoring results and output the monitoring results to the front-end interface;

[0012] The first monitoring process includes:

[0013] Generate library table data according to the task information;

[0014] Use the data quality detection rules corresponding to each data governance task in PostgreSQL to detect the database and table data;

[0015] If the database table data does not meet the data quality detection rules, blocking processing is performed on the data governance task corresponding to the database table data;

[0016] The second monitoring process includes:

[0017] Analyze whether the acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data meet the alarm rules;

[0018] The monitoring indicators that meet the alarm rules are output through the alarm manager.

[0019] Preferably, the step of obtaining task information of data governance tasks, acquisition program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the data governance platform, virtual machine cluster, and kubernetes cluster includes:

[0020] Send a request to the query interface of each data governance task in the data governance platform, virtual machine cluster, and kubernetes cluster;

[0021] Receive task information of each data governance task based on feedback from the acquisition request;

[0022] Obtain collection program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the push gateway.

[0023] Preferably, after obtaining task information of the task to be monitored, collecting program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the data governance platform and the Kubernetes cluster, the process further includes:

[0024] generating a chart based on the acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data;

[0025] The chart is displayed through an analysis and visualization platform.

[0026] Preferably, before analyzing whether the acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data meet the alarm rules, the method further includes:

[0027] The acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data are stored in a log monitoring and alarm component.

[0028] Preferably, before obtaining the collection program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data from the push gateway, the process further includes:

[0029] Receive custom business monitoring indicators and collection process indicators sent by the collection program through the push gateway;

[0030] Marking the custom business monitoring indicator and the collection process indicator as collection program monitoring data;

[0031] Receive custom business monitoring indicators and business indicators sent by the offline data warehouse through the push gateway, and mark them as offline data warehouse program monitoring data;

[0032] The custom business monitoring indicators and business indicators sent by the real-time data warehouse are received through the push gateway and marked as real-time data warehouse program monitoring data.

[0033] A second aspect of the present invention discloses a full-link data monitoring device, comprising:

[0034] The acquisition unit is used to obtain task information of data governance tasks, collection program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the data governance platform, virtual machine cluster, and Kubernetes cluster;

[0035] A first execution unit, configured to execute a first monitoring process for the task information of the data governance task;

[0036] a second execution unit, configured to execute a second monitoring process for the acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data;

[0037] An output unit, configured to output monitoring results and output the monitoring results to a front-end interface;

[0038] The first monitoring process includes:

[0039] A generating module, configured to generate library table data according to the task information;

[0040] A detection module, configured to detect the database and table data using the data quality detection rules corresponding to each of the data governance tasks in PostgreSQL;

[0041] A blocking processing module is used to perform blocking processing on the data governance task corresponding to the library table data if the library table data does not meet the data quality detection rules;

[0042] The second monitoring process includes:

[0043] An analysis module, configured to analyze whether the acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data comply with alarm rules;

[0044] The output module is used to output monitoring indicators that meet the alarm rules through the alarm manager.

[0045] Preferably, the acquisition unit includes:

[0046] The sending module is used to send acquisition requests to the query interface of each data governance task in the data governance platform, virtual machine cluster, and kubernetes cluster;

[0047] A receiving module, configured to receive task information of each data governance task based on feedback of the acquisition request;

[0048] The acquisition module is used to obtain the collection program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the push gateway.

[0049] Preferably, the device further comprises:

[0050] a generating unit, configured to generate a chart based on the collected program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data;

[0051] A display unit is used to display the chart through an analysis and visualization platform.

[0052] Preferably, the device further comprises:

[0053] The storage unit is used to store the acquisition program monitoring data, the offline data warehouse program monitoring data and the real-time data warehouse program monitoring data in the log monitoring and alarm component.

[0054] Preferably, the device further comprises:

[0055] The first receiving unit is used to receive the custom business monitoring indicators and collection process indicators sent by the collection program through the push gateway;

[0056] a marking unit, configured to mark the user-defined business monitoring indicator and the collection process indicator as collection program monitoring data;

[0057] A second receiving unit is configured to receive, through the push gateway, custom business monitoring indicators and business indicators sent by the offline data warehouse, and mark them as offline data warehouse program monitoring data;

[0058] The third receiving unit is configured to receive the custom business monitoring indicators and business indicators sent by the real-time data warehouse through the push gateway, and mark them as real-time data warehouse program monitoring data.

[0059] Based on the above-mentioned embodiment of the present invention, a full-link data monitoring method and device are provided, which obtains the task information of the data governance task, the acquisition program monitoring data, the offline data warehouse program monitoring data and the real-time data warehouse program monitoring data from the data governance platform, the virtual machine cluster and the kubernetes cluster; for the task information of the data governance task, the first monitoring process is executed; for the acquisition program monitoring data, the offline data warehouse program monitoring data and the real-time data warehouse program monitoring data, the second monitoring process is executed. Problems in the data link are identified through real-time anomaly detection. Adopting closed-loop quality management, the integrity, consistency and accuracy of the data are continuously monitored to form a closed loop of problem discovery, repair and verification, and gradually improve the data quality. At the same time, the complete link of the data from the source to the application is tracked, the data source, processing logic and usage scenarios are clarified, and ultimately the compliance, security, quality and efficiency of the data in each link are ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0061] Figure 1 A schematic diagram of the architecture of a full-link data monitoring system provided by an embodiment of the present invention;

[0062] Figure 2 A flowchart of a full-link data monitoring method provided by an embodiment of the present invention;

[0063] Figure 3 An example diagram of obtaining monitoring data provided by an embodiment of the present invention;

[0064] Figure 4 An example diagram of a monitoring data governance task provided by an embodiment of the present invention;

[0065] Figure 5 This is a structural block diagram of a full-link data monitoring device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0067] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0068] As can be seen from the background technology, traditional manual management is difficult to fully monitor the data life cycle, lacks flexibility, and cannot cope with complex needs. Manual intervention increases costs and brings data leakage, quality degradation and compliance risks.

[0069] Therefore, an embodiment of the present invention provides a full-link data monitoring method and device, which obtains task information of data governance tasks, acquisition program monitoring data, offline data warehouse program monitoring data and real-time data warehouse program monitoring data from the data governance platform, virtual machine cluster and kubernetes cluster; for the task information of the data governance task, executes a first monitoring process; for the acquisition program monitoring data, the offline data warehouse program monitoring data and the real-time data warehouse program monitoring data, executes a second monitoring process. Identify problems in the data link through real-time anomaly detection. Adopt quality closed-loop management, continuously monitor the integrity, consistency and accuracy of the data, form a closed loop of problem discovery, repair and verification, and gradually improve data quality. At the same time, track the complete link of data from the source to the application, clarify the data source, processing logic and usage scenarios, and ultimately ensure the compliance, security, quality and efficiency of the data in each link.

[0070] See also Figure 1 , shows a schematic diagram of the architecture of a full-link data monitoring system provided by an embodiment of the present invention. The system includes: a monitoring system, Prometheus, a logging monitoring alert (LMA) component, and a registration center (Consul).

[0071] It should be noted that this full-link data monitoring system is deployed in a Kubernetes cluster. The system is used to monitor the operating system, middleware, and applications that are part of the Kubernetes cluster, VM cluster, and data governance platform.

[0072] Among them, the monitoring system at least includes a monitoring system front end, a monitoring system alarm module, a monitoring system back end, a service dialing module, and an analysis and visualization platform.

[0073] Prometheus is an open source monitoring and alerting system specifically designed for collecting, storing, and querying time series data. It includes at least the AlertManager, Prometheus Center, and Prometheus k8s.

[0074] The log monitoring and alarm component LMA integrates basic components such as telegraf, node-exporter, influxdb, ELK, etc., and is used to collect logs and monitoring data of all components on the k8s cluster (including all components of the data governance platform).

[0075] The registration center (Consul) is responsible for the service registration of monitoring plug-ins (Telegraf, Exporter). Prometheus obtains the registration information of monitoring plug-ins through Consul.

[0076] like Figure 1 As shown in the figure, the monitoring system is connected to the alarm manager through the monitoring system backend. The alarm manager is connected to Prometheus Center. Prometheus Center is connected to Prometheus Kubernetes. Prometheus Kubernetes is connected to the log monitoring and alarm component. The log monitoring and alarm component is connected to the data governance platform. Prometheus Kubernetes is connected to the VM cluster and the registration center (Consul).

[0077] It is understandable that the data governance platform is deployed on a Kubernetes cluster and includes a complete set of governance tools such as management center, data integration, data development, data assets, data standards, data modeling, data quality, data security, data services, data visualization, data sources and data labels, aiming to achieve full life cycle management of data from collection, processing to use.

[0078] It should be noted that monitoring nodes (agents) should be deployed in advance on the nodes that need to be monitored in the Kubernetes cluster, VM cluster, and data governance platform, and information should be collected using the monitoring nodes. Types of monitoring nodes include, but are not limited to, Telegraf (agents), cloud-native monitoring tools, and custom exporters.

[0079] Combine Figure 1The analysis and visualization platform shown in the monitoring system is Grafana, which is primarily used to monitor, analyze, and display time series data. It can connect to a variety of data sources, such as Prometheus, InfluxDB, Elasticsearch, MySQL, and PostgreSQL, and graphically display data through dashboards, helping to monitor and analyze system status in real time.

[0080] It should be noted that the monitoring system connects to the third-party message notification interface through the AlertManager.

[0081] As you can understand, AlertManager is part of the Prometheus ecosystem and is specifically designed to process and manage alerts. Its main function is to receive alert information from Prometheus Center, group these alerts, deduplicate, suppress, route them, and ultimately send them to designated notification channels (such as email).

[0082] It should be noted that Prometheus Center regularly pulls monitoring data from Prometheus k8s for analysis.

[0083] Specifically, PostgreSQL is responsible for storing configuration information such as alarm rules, clusters, services, and monitoring registration. The InfluxDB component within the Log Monitoring and Alerting (LMA) component is responsible for storing monitoring metrics, with a data retention policy of 180 days. These metrics include server performance indicators (such as CPU utilization, memory utilization, and disk utilization), application component indicators (such as the current number of Nginx client connections and client requests), and custom business indicators (such as data processing timeliness and data latency).

[0084] It is understandable that the persistent monitoring data of Prometheus k8s is stored in the LMA's influxdb time series database.

[0085] In actual applications, the monitoring system interacts with Prometheus, AlertManager, and Consul components, specifically: interacting with Prometheus to query monitoring indicators and the health status of Targets; interacting with AlertManager to configure alarm rules and notification methods; and interacting with Consul to configure monitoring plug-in information.

[0086] It can be understood that the service dialing module in the monitoring system is used to provide service dialing functions, perform health checks on various services on the data governance platform, and check the survival status of the services.

[0087] Combine Figure 1 As shown in the following example, within the same network domain, Prometheus directly pulls monitoring indicators from the virtual cluster (such as Figure 1 As shown in the virtual machine cluster (A) Prometheus Traget); in the case of different network domains, it is necessary to deploy a sub-Prometheus in an isolated network (such as Figure 1 The Prometheus Child is shown in the figure), and the network policy between the full-link data monitoring system and the child Prometheus is opened. The child Prometheus collects the performance indicators of the monitoring plug-ins in the virtual cluster (such as Figure 1 The virtual machine cluster (B) Prometheus Traget is shown, and then the monitoring system Prometheus pulls the monitoring data aggregated by the sub-Prometheus.

[0088] Combined with the above content, this full-link data monitoring system can realize full-link data monitoring of data in the data governance platform, virtual machine cluster and kubernetes cluster. For specific implementation methods, see Figure 2 , shows a flow chart of a full-link data monitoring method provided by an embodiment of the present invention. The method includes:

[0089] Step S201: Obtain task information of data governance tasks, collection program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the data governance platform, virtual machine cluster, and kubernetes cluster.

[0090] In the specific implementation of step S201, Prometheus is used to obtain task information of data governance tasks, collection program monitoring data, offline data warehouse program monitoring data and real-time data warehouse program monitoring data from the data governance platform, virtual machine cluster and kubernetes cluster.

[0091] Specifically, the acquisition process is as follows (process A1 to process A3):

[0092] Process A1: Send a request to the query interface of each data governance task in the data governance platform, virtual machine cluster, and kubernetes cluster.

[0093] In the specific implementation of process A1, since the query interface (such as the metrics interface) of each data governance task in the data governance platform, virtual machine cluster, and Kubernetes cluster is connected to Prometheus, Prometheus is used to send a query request to the query interface of each data governance task in the data governance platform, virtual machine cluster, and Kubernetes cluster.

[0094] It should be noted that data governance tasks include but are not limited to Flink real-time tasks, DolphinScheduler processes and tasks, data development tasks, and data exchange tasks.

[0095] Flink real-time tasks refer to real-time data processing tasks executed in the streaming data processing framework (Apache Flink). DolphinScheduler processes refer to the execution order and dependencies of a series of tasks defined in the workflow scheduling system (DolphinScheduler). DolphinScheduler tasks refer to specific units of work defined in the workflow scheduling system.

[0096] Process A2: Receive task information for each data governance task based on the request feedback.

[0097] During the specific implementation of process A2, Prometheus is used to receive task information based on request feedback for each data governance task.

[0098] It can be understood that based on the task information, the task status of each data governance task can be viewed.

[0099] Process A3: Obtain the collected program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the push gateway.

[0100] It should be noted that in addition to monitoring data governance tasks, the embodiments of the present invention also monitor key task points in the data link, such as real-time monitoring of the running status of the collection program, offline data warehouse program and real-time data warehouse program.

[0101] Specifically, real-time monitoring is achieved by analyzing and collecting program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data. Figure 3 The specific process of obtaining the collected program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data is as follows (process B1 to process B4):

[0102] Process B1: Receive custom business monitoring indicators and collection process indicators sent by the collection program through the push gateway.

[0103] As you can understand, the push gateway is specifically PushGateway. PushGateway is a component in the Prometheus ecosystem that is primarily used to receive measurement data from short-running jobs (such as batch tasks). It allows these jobs to "push" indicators to Prometheus.

[0104] In the specific implementation process B1, the custom business monitoring indicators and collection process indicators sent by the collection program are received based on PushGateway.

[0105] like Figure 3 As shown in the figure, for custom business metrics (such as collected data volume and collection latency), the system launches a separate monitoring thread for each collection program, collects monitoring data in real time, and sends it to Kafka. Flink then consumes the monitoring data from Kafka, transforms it, and converts it into custom business monitoring metrics. These custom business metric data are then pushed to Prometheus in the monitoring system via PushGateway.

[0106] To monitor and collect process metrics, we use telegraf-jolokia to collect status information for each Java process, including class loading status, memory usage, thread management, and garbage collection status. The collected metrics are sent to Kafka. Flink then consumes this monitoring data in real time and pushes it to Prometheus in the monitoring system through PushGateway for storage and management.

[0107] Process B2: Mark the custom business monitoring indicators and collection process indicators as collection program monitoring data.

[0108] When specifically implementing process B2, the custom business monitoring indicators and collection process indicators pushed by PushGateway are marked as collection program monitoring data.

[0109] Process B3: Receive custom business monitoring indicators and business indicators sent by the offline data warehouse through the push gateway and mark them as offline data warehouse program monitoring data.

[0110] In the specific implementation process B3, the custom business monitoring indicators and business indicators sent by the offline data warehouse are received based on PushGateway, and then the custom business monitoring indicators and business indicators are marked as offline data warehouse program monitoring data.

[0111] Process B4: Receive custom business monitoring indicators and business indicators sent by the real-time data warehouse through the push gateway and mark them as real-time data warehouse program monitoring data.

[0112] In the specific implementation process B4, the custom business monitoring indicators and business indicators sent by the real-time data warehouse are received based on PushGateway, and then the custom business monitoring indicators and business indicators are marked as offline data warehouse program monitoring data.

[0113] Combine Figure 3As shown, Spark / Flink programs (i.e. offline / real-time data warehouses) pass PushGateway (i.e. Figure 3 Flink-metrics and Spark-metrics shown in the figure) connect the monitoring data of custom business indicators to the monitoring system Prometheus. During the operation of Spark / Flink tasks, the tasks will push business indicators to PushGateway (i.e. Figure 3 As shown in Flink-metrics and Spark-metrics, Prometheus will regularly pull these monitoring data from PushGateway.

[0114] In addition, Spark / Flink's own monitoring indicators are connected to the Prometheus monitoring system through Exporter.

[0115] In some specific embodiments, charts are generated based on the collected program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data; and the charts are displayed through an analysis and visualization platform (Grafana).

[0116] Step S202: Execute a first monitoring process for the task information of the data governance task.

[0117] It should be noted that the first monitoring process includes (process C1 to process C3):

[0118] It is understandable that, combined with Figure 4 The content shown here indicates that data quality modules (such as Prometheus) provide SDKs for Flink to call. While running tasks, Flink calls methods in the data quality SDK and sends task information to the data quality module.

[0119] Process C1: Generate library table data based on task information.

[0120] It should be noted that, for example Figure 4 The displayed task information includes but is not limited to the Flink task ID, Flink task target table name, Flink task status (Created, Running, Finished, Cancelling, Cancelled, Restarting, Failing, Failed, Suspended), task type (Flink, Spark), and job platform (Dinky, DolphinScheduler).

[0121] It is understandable that the library table data is configured based on these task information, such as configuring the target table of the Flink task (such as Figure 4 This is equivalent to registering Flink task information in the data quality module. Once an anomaly is detected, the associated Flink task will be interrupted.

[0122] Process C2: Use the data quality detection rules corresponding to each data governance task in PostgreSQL to detect the database and table data.

[0123] Specifically, we use the data quality detection rules pre-written in PostgreSQL for each data governance task to check the database and table data (for example Figure 4 target table shown in the figure) for detection.

[0124] Process C3: If the database table data does not meet the data quality detection rules, the data governance task corresponding to the database table data is blocked.

[0125] It is understandable that when the database and table data do not comply with the data quality detection rules, the relevant data governance tasks will be effectively blocked to prevent the spread of erroneous data and ensure the overall data quality of the system.

[0126] Step S203: Execute the second monitoring process for the collected program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data.

[0127] It can be understood that the second monitoring process includes (process D1 and process D2):

[0128] Process D1: Analyze the collected program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data to see if they meet the alarm rules.

[0129] It should be noted that alarm rules are pre-stored in PostgreSQL. Threshold alarms (such as data delay timeouts and spikes in task failure rates) can be set in these rules to prevent minor issues from escalating into systemic risks.

[0130] In some specific embodiments, the analysis program stores the collected program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data (such as monitoring data that requires in-depth analysis) in the log monitoring and alarm component, that is, persists them in InfluxDB.

[0131] Process D2: Output the monitoring indicators that meet the alarm rules through the alarm manager.

[0132] Step S204: Output the monitoring results and output the monitoring results to the front-end interface.

[0133] In the specific implementation of step S204 , the monitoring results are output (as shown in process C3 and process D2 ), and then the monitoring results are displayed through the front-end interface.

[0134] In some specific embodiments, the full-link dependency graph can also be used to quickly locate the fault point (for example, upstream data delay causes downstream task failure), thereby effectively shortening the mean time to repair (MTTR).

[0135] In the embodiments of the present invention, by covering the entire life cycle of data from generation to consumption, data quality can be effectively improved, data credibility can be enhanced, and compliance and audit requirements can be met. Real-time anomaly detection automatically identifies anomalies (such as missing data, format errors, logical contradictions, etc.) in data collection, processing, and storage, reducing the impact of dirty data on downstream processes. Through closed-loop quality management, indicators such as integrity, consistency, and accuracy are continuously monitored to form a closed loop for discovering, repairing, and verifying data quality issues, gradually improving data quality. At the same time, the complete link of data from source to application is tracked, clarifying the data source, processing logic, and usage scenarios, ensuring that each link is "visible, measurable, and controllable," achieving a shift from "post-event firefighting" to "pre-event prevention and in-event intervention," ultimately ensuring the compliance, security, quality, and efficiency of data at every stage.

[0136] Corresponding to the full-link data monitoring method provided by the above embodiment of the present invention, see Figure 5 , shows a structural block diagram of a full-link data monitoring device provided by an embodiment of the present invention.

[0137] The device includes: an acquisition unit 501 , a first execution unit 502 , a second execution unit 503 and an output unit 504 .

[0138] The acquisition unit 501 is used to obtain task information of data governance tasks, collection program monitoring data, offline data warehouse program monitoring data and real-time data warehouse program monitoring data from the data governance platform, virtual machine cluster and kubernetes cluster.

[0139] The first execution unit 502 is used to execute a first monitoring process for task information of the data governance task.

[0140] The second execution unit 503 is configured to execute a second monitoring process for the collected program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data.

[0141] The output unit 504 is used to output the monitoring results and output the monitoring results to the front-end interface.

[0142] The first monitoring process includes:

[0143] The generation module is used to generate library table data based on task information.

[0144] The detection module is used to detect database and table data using the data quality detection rules corresponding to each data governance task in PostgreSQL.

[0145] The blocking processing module is used to perform blocking processing on the data governance tasks corresponding to the library table data if the library table data does not meet the data quality detection rules.

[0146] The second monitoring process includes:

[0147] The analysis module is used to analyze whether the collected program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data meet the alarm rules.

[0148] The output module is used to output monitoring indicators that meet the alarm rules through the alarm manager.

[0149] In the embodiments of the present invention, by covering the entire life cycle of data from generation to consumption, data quality can be effectively improved, data credibility can be enhanced, and compliance and audit requirements can be met. Real-time anomaly detection automatically identifies anomalies (such as missing data, format errors, logical contradictions, etc.) in data collection, processing, and storage, reducing the impact of dirty data on downstream processes. Through closed-loop quality management, indicators such as integrity, consistency, and accuracy are continuously monitored to form a closed loop for discovering, repairing, and verifying data quality issues, gradually improving data quality. At the same time, the complete link of data from source to application is tracked, clarifying the data source, processing logic, and usage scenarios, ensuring that each link is "visible, measurable, and controllable," achieving a shift from "post-event firefighting" to "pre-event prevention and in-event intervention," ultimately ensuring the compliance, security, quality, and efficiency of data at every stage.

[0150] Combine Figure 5 The content shown, the acquisition unit 501, includes: a sending module, a receiving module and an acquisition module.

[0151] The sending module is used to send acquisition requests to the query interface of each data governance task in the data governance platform, virtual machine cluster, and kubernetes cluster.

[0152] The receiving module is used to receive task information of each data governance task based on the request feedback.

[0153] The acquisition module is used to obtain the collection program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the push gateway.

[0154] Combine Figure 5 As shown in the content, the device also includes: a generating unit and a display unit.

[0155] The generation unit is used to generate charts based on the collected program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data.

[0156] The display unit is used to display charts through the analysis and visualization platform.

[0157] Combine Figure 5 As shown, the device also includes: a storage unit for storing the collected program monitoring data, offline data warehouse program monitoring data and real-time data warehouse program monitoring data in the log monitoring and alarm component.

[0158] Combine Figure 5 As shown in the content, the device further includes: a first receiving unit, a marking unit, a second receiving unit and a third receiving unit.

[0159] The first receiving unit is configured to receive, through a push gateway, user-defined business monitoring indicators and collection process indicators sent by a collection program.

[0160] The marking unit is used to mark the custom business monitoring indicators and collection process indicators as collection program monitoring data.

[0161] The second receiving unit is used to receive the custom business monitoring indicators and business indicators sent by the offline data warehouse through the push gateway, and mark them as offline data warehouse program monitoring data.

[0162] The third receiving unit is used to receive the custom business monitoring indicators and business indicators sent by the real-time data warehouse through the push gateway, and mark them as real-time data warehouse program monitoring data.

[0163] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0164] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0165] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A full-link data monitoring method, characterized in that: The method comprises: Obtain task information for data governance tasks, collection program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the data governance platform, virtual machine cluster, and Kubernetes cluster. For the task information of the data governance task, executing a first monitoring process; executing a second monitoring process for the acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data; Output monitoring results and output the monitoring results to the front-end interface; The first monitoring process includes: Generate library table data according to the task information; Use the data quality detection rules corresponding to each data governance task in PostgreSQL to detect the database and table data; If the database table data does not meet the data quality detection rules, blocking processing is performed on the data governance task corresponding to the database table data; The second monitoring process includes: Analyze whether the acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data meet the alarm rules; The monitoring indicators that meet the alarm rules are output through the alarm manager.

2. The method according to claim 1, characterized in that The task information of data governance tasks, collection program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data are obtained from the data governance platform, virtual machine cluster, and Kubernetes cluster, including: Send a request to the query interface of each data governance task in the data governance platform, virtual machine cluster, and kubernetes cluster; Receive task information of each data governance task based on feedback from the acquisition request; Obtain collection program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the push gateway.

3. The method according to claim 1, characterized in that After obtaining task information of the monitored tasks, collecting program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the data governance platform and Kubernetes cluster, it also includes: generating a chart based on the acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data; The chart is displayed through an analysis and visualization platform.

4. The method according to claim 1, wherein Before analyzing whether the acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data meet the alarm rules, the method further includes: The acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data are stored in a log monitoring and alarm component.

5. The method according to claim 2, characterized in that Before obtaining the collection program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the push gateway, the following is also included: Receive custom business monitoring indicators and collection process indicators sent by the collection program through the push gateway; Marking the custom business monitoring indicator and the collection process indicator as collection program monitoring data; Receive custom business monitoring indicators and business indicators sent by the offline data warehouse through the push gateway, and mark them as offline data warehouse program monitoring data; The custom business monitoring indicators and business indicators sent by the real-time data warehouse are received through the push gateway and marked as real-time data warehouse program monitoring data.

6. A full-link data monitoring device, characterized in that: The device comprises: The acquisition unit is used to obtain task information of data governance tasks, collection program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the data governance platform, virtual machine cluster, and Kubernetes cluster; A first execution unit, configured to execute a first monitoring process for the task information of the data governance task; a second execution unit, configured to execute a second monitoring process for the acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data; An output unit, configured to output monitoring results and output the monitoring results to a front-end interface; The first monitoring process includes: A generating module, configured to generate library table data according to the task information; A detection module, configured to detect the database and table data using the data quality detection rules corresponding to each of the data governance tasks in PostgreSQL; A blocking processing module is used to perform blocking processing on the data governance task corresponding to the library table data if the library table data does not meet the data quality detection rules; The second monitoring process includes: An analysis module, configured to analyze whether the acquisition program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data comply with alarm rules; The output module is used to output monitoring indicators that meet the alarm rules through the alarm manager.

7. The device according to claim 6, characterized in that The acquisition unit includes: The sending module is used to send acquisition requests to the query interface of each data governance task in the data governance platform, virtual machine cluster, and kubernetes cluster; A receiving module, configured to receive task information of each data governance task based on feedback of the acquisition request; The acquisition module is used to obtain the collection program monitoring data, offline data warehouse program monitoring data, and real-time data warehouse program monitoring data from the push gateway.

8. The device according to claim 6, characterized in that The device further comprises: a generating unit, configured to generate a chart based on the collected program monitoring data, the offline data warehouse program monitoring data, and the real-time data warehouse program monitoring data; A display unit is used to display the chart through an analysis and visualization platform.

9. The device according to claim 6, characterized in that The device further comprises: The storage unit is used to store the acquisition program monitoring data, the offline data warehouse program monitoring data and the real-time data warehouse program monitoring data in the log monitoring and alarm component.

10. The device according to claim 7, characterized in that The device further comprises: The first receiving unit is used to receive the custom business monitoring indicators and collection process indicators sent by the collection program through the push gateway; a marking unit, configured to mark the user-defined business monitoring indicator and the collection process indicator as collection program monitoring data; A second receiving unit is configured to receive, through the push gateway, custom business monitoring indicators and business indicators sent by the offline data warehouse, and mark them as offline data warehouse program monitoring data; The third receiving unit is configured to receive the custom business monitoring indicators and business indicators sent by the real-time data warehouse through the push gateway, and mark them as real-time data warehouse program monitoring data.

Images